Guidance from Data Ethics Advisory Group

We have collated the guidance provided by the Data Ethics Advisory Group (DEAG) to government agencies who have brought specific initiatives and challenges to the Group. The guidance captures and themes advice given in response to these specific requests, however, much of the guidance will be generalisable to many use cases.

Overarching Considerations before a Decision to Proceed

Consider use cases individually and be clear about what is in and out of scope. Describe each use case and consider the ethics relating to each. Learn from others e.g., international initiatives.

Consider all actual and perceived risks

Note the foundational risks of trust, confidence, and engagement with the data system (when the public trusts government with their data, good data quality results, strengthening official statistics)
Note that data continues to have a life of its own once it is shared
Remember that data released publicly can then be used commercially
Ask questions like - could disadvantaged populations be further disadvantaged by initiatives or interventions?
What level of residual risk is acceptable?

Consider the potential benefits

What use is the data being put to?
Who may benefit from the data use and application?
Are the benefits being returned to the public of New Zealand?
Will this foster local innovation - or are the benefits going overseas or to commercial entities? If entities are going to make a lot of money overseas, what commercial gains would need to flow back into New Zealand?
What meaningful impact will eventuate?
Measure where the impacts may occur over time while recognising that communities' needs may change over time.

Consider equity, inclusion and reciprocity

A digital divide is real – will this exacerbate it? How could digital inclusion occur?
Is upskilling of capability needed?
Promote accessibility e.g., give people access to the data about them and their communities
Reciprocate where possible and give data back to communities
The environment and future generations also have rights
Use an ethical framework, for example:
- Consequentialism
- Utility and fairness
- Rights and justices.

Consider the balance of benefits and potential harms to reach a final decision

Follow the principle of ‘doing good while doing no harm’. Protect people and communities.
It is important to reflect on what is possible and what is necessary.
Carefully consider ‘should we do this?
Consider what are the ethics of not doing this?
When risk and sensitivities are involved, there is a need to proceed very carefully.

Data – in general

Some data and data practices require certain ethical considerations

Sensitive data: data can be considered sensitive by different people, cultures, and communities e.g., Māori data that is tapu, rainbow communities who are fearful to self-identify, imaging, cervical screening. How should this data be treated?
Ethnicity data: ethnicity goes to the heart of peoples’ identity; therefore, it is important that individuals and communities can define ethnicity on their terms. Be aware that some people do not feel safe in self-identifying their ethnicity in certain situations, fearing that they may be at risk of negative outcomes, so they may report ethnicity differently in different contexts. Ethnicity can also change over time.
Missing data: it is important to understand who is not in the data. These individuals and communities are not represented and not visible. How can the data be more inclusive?
Imputation: consider human rights and autonomy issues when imputing missing data and the value of non-responses.
Administrative data: consider how data was collected (e.g., was information provided by the individual or someone else? was it collected under duress or stress?) when considering whether it is appropriate to use or not. Note that people can give their information (e.g., ethnicity) in different ways to different places. There are many factors that affect the quality of administrative data that need to be considered.
Synthetic data: when synthetic data closely models real populations, carefully consider any privacy concerns.
Proxy measures: be cautious of using data attributes as proxy measures as these can prove to be poor substitutes and can introduce inaccuracies and bias.
Inferences: inferences can also result in harm, such as discrimination and stigma.
Simulation modelling: there are often trade-offs to be made when creating models. Careful thought is needed around what data and attributes are used, including sampling methodology that ensures representativeness.

Data Collection

When collecting data from people it is important to allow for autonomy and to build trust

Use prior informed consent that uses clear language and a high level of transparency. Messaging should be user-friendly and informative.
Allow autonomy and choice by providing options and giving people the ability to not answer a question.
Don’t over-collect more data than is needed for the existing purpose.
Carefully consider the choice of sampling methodology and recruitment to ensure representation of those who have not previously been well included in the data.
Consider things such as - if one person in the household shares their data, how could this impact on the household or group of households/groups they belong to?
Consider the attributes to be collected to allow for visibility and intersectionality e.g., avoid binary choices such as Māori and non-Māori.
Creating diverse and representative datasets in research will contribute to more accurate insights, and ultimately, better outcomes for all.

Data Sharing

Data sharing, especially with those who the data is from, can empower communities and enable benefits for New Zealanders, but sharing data needs to be done carefully to ensure that privacy and ethics are maintained.

It is important to consider equity, transparency, use purpose, and data minimisation when sharing data

Consider issues of equity and fairness when allowing access to data.
Be transparent on what data is being shared, to who, and for what purpose.
The purpose the data will be used for should align with the purpose of the initial data collection.
If this data is being shared with commercial entities, what benefit is flowing back to those whose data it is?
Only share the minimal amount of data necessary, at the minimum level of data detail.

Where the data is personal information, ensure protection of privacy and confidentiality – for both individuals and groups

The risk of re-identification:

There is increasing ability to re-identify information and greater capability to reverse engineer de-identification processes (e.g., hashing).
The risk of re-identification is greater when bringing together data from different data sources.

Consider technical ways of sharing information that minimise the risk of identifying individuals and groups, for example:

Technical approaches that preserve privacy e.g. homomorphic encryption (a form of encryption that allows computations to be performed on encrypted data without first having to decrypt it).
There are options to be more permissible with how the data is used, while maintaining protective control of the data e.g., receiving queries from entities and then releasing the aggregated output of those queries, not the data itself.
Federated data models allow agencies and groups to retain control over their data and what data is shared.
Temporary links to data can be used to tightly manage access.
Differentiated privacy settings can enable entities to access different levels of data.
Data tags can protect some data for specific use, while other data may be made accessible more readily.

Engagement

Engage with groups and communities in ways that build trust and value people

Engage early and build trust.
Consider how to bring the public on the journey, to increase trust and confidence.
Be transparent.
Language is important. Consider the content and tone.
Design for accessibility and inclusivityg., translate consultation documents into other languages and allow people to make verbal submissions.
Consider appropriate reciprocity for people’s time and expertise.

Engage with key demographics and priority populations

Engage with Māori and design specific engagement with mana whenua.
Conduct more targeted engagement with those likely to be most impacted. These are the people you most need to hear from and where engagement effort should be targeted, rather than engaging with those who are easy to find.
Engage with those who are traditionally misrepresented, under-represented or not present in the data (this is especially true for administrative data sources).
We recommend a focus on working with minorities, with a rationale that what works well for minority groups will work well for the majority of people. For example, if products are developed that meet the specific needs of people with a disability, usability will be improved for everyone.
Consider engaging with rainbow, disabled, Pacific, ethnic minorities, communities that have low trust in Government, etc

Authorising Environment

It is important to operate within the right ethical and legal frameworks and explicitly express how decisions align with these. Identify possible consequences for data to be used in ways that could lead to harm/s, or in ways that could extinguish rights.

Consider the legislative and ethical environment

Data sovereignty rights.
Jurisdictional risks e.g. data held in, or shared with, other countries. Jurisdictional risks refer to the legal pathway for foreign governments, for example Australia, to access NZ data held in their jurisdictions.
Territorial reach is also legally possible, under current mutual assistance laws, but has not yet been experienced. This is where other governments, in certain circumstances, can request access to data held in NZ.
Privacy Law and whether engagement with the Office of the Privacy Commissioner is advisable.
Human Rights.
The rights of the environment and future generations.

Seek to maintain and build social license, trust and confidence

Align to trust criteria and fully incorporate the classical philosophical structure of trust.
Put in place strong guardrails and frameworks to ensure government data use does not overreach.
Be aware of power imbalances e.g., The role of Stats NZ as one of the few agencies that people must give information to, and the recognition of its constitutional role.

Develop strong governance

Ensure there are clear accountabilities for data.
Consider an independent monitoring function to ensure the agency is following necessary assurance rules. This may alleviate public concerns and provide reassurance to ministers.
Ensure the proposed use of the data aligns with the original purpose of data collection. For the individuals that the data relates to, would they reasonably expect it to be used in this way?

Artificial Intelligence (AI) specific considerations

Moral responsibility is needed in this space where people’s lives are impacted. Technology can be misused where ‘what is a tool to someone can be a weapon to someone else’. Data standards, data quality, data governance, data protection, and privacy are of critical importance.

Responsible AI Governance is needed

Use a responsible AI Governance framework
Conduct an AI Impact Assessment (AIA)
Monitor downstream effects.

Accuracy and Reliability

Accuracy: Consider accuracy in relation to AI, both in terms of timeliness and veracity. Check all information before it is relied on. Test and report on the accuracy of data. An ability to tag and preserve the original base data is needed.
Hallucinations: Clearly define ‘hallucinations’ and call these out as a key risk.
Fake information: Generative AI has the potential to create fake information at scale. This is a key concern to be aware of.

Consider risks associated with security, privacy and potential overreach

Privacy by design is critical along with knowing where data is stored and retained – could it be open to misuse?
Security concerns exist, including:
- the potential for adversarial attacks to identify training data
- the potential to train the AI model to act in ways it was not meant to
- the rule of law within the countries that overseas suppliers may be operating out of, and the history of response by the suppliers to issues of jurisdiction, and how they might protect New Zealand organisations.
The potential for profiling and surveillance capabilities of a system, including its ability to build a comprehensive picture of a person’s movements (e.g., Automatic Number Plate Recognition) and the privacy risks associated with this, along with impacts on trust and confidence in Government.

Assess for potential bias and aim for equity

Training data: It is important to understand the data that an AI model is trained on. Models trained on overseas data (e.g., most ‘off the shelf’ offerings) will need to be customised for the New Zealand context by the inclusion of relevant training data. Clarity of ownership over this training data is also essential. High impact areas like the NZ Health system, especially need data that reflects the diversity of New Zealanders. Facial recognition AI is a known case where the composition of the training data does not reflect NZ demographics and consequently the AI has the potential to not recognise certain NZ demographics, leading to false positives.
Māori and Māori worldview: Generative AI doesn’t presently cater to a Māori consideration of data which means that inherent biases exist in AI against Māori and Māori worldview. There are no checks that data is accurate or authentic. Cultural differences need to be acknowledged or could lead to cultural harm.
Automation bias: Automation bias is the human tendency to rely on machine outputs, often at the expense of their own judgement. For example, it appears there may have been automation bias at play in the recent Rotorua case[1] where facial recognition technology used by Foodstuffs North Island Limited incorrectly identified a person as a shoplifter. There is a need for agencies to appropriately train the staff involved in monitoring AI decisions.
Representation in the data: The risks surrounding AI may not be shared equally amongst all citizens and populations, for example, those that may be personally identifiable within data sets, or those that may not be within any data sets.

Investigate data provenance and protect against cultural appropriation or breaches to copyright

There are risks associated with cultural appropriation where generative AI may inappropriately use mātauranga Māori, which has an intellectual property part and also a cultural responsibility part.
Be authentic and treat the information with integrity; meaning an acknowledgement that the data belongs to the source, rather than the collector or repository of the data.

Ensure transparency and explain decisions made when using AI tools.

Government automated decision-making can significantly impact the public. There is a need for people to be able to understand how those decisions are made so they can challenge decisions that unfairly impact them.
Consider carefully which decisions need to be made by a human.

Provide training and resources to lift the capability to work with AI

Provide training opportunities for staff so they understand that generative AI is a sophisticated predictive text system, and how best to use it, especially with sensitive data.
AI skills training and awareness are needed for executives and leadership teams and those workforces where there are going to be significant changes due to AI and automation. Different business sectors may require different support.
Resources are needed to demystify AI and engage people, with a focus on data ethics.

AI Procurement considerations

There are some key differences in procuring AI and other technologies, and smaller government departments rely heavily on larger departments in undertaking due diligence as part of their procurement processes.

As the Government is the largest technology procurer in New Zealand, there is also an opportunity to influence and encourage appropriate AI governance by suppliers through the procurement process. For example, requiring vendors to demonstrate how they are taking steps to manage accuracy, privacy, bias and explainability risks will help encourage a greater focus on minimising the well-established risks of AI.

Given New Zealand’s size and the common issues faced by all agencies (e.g., the need for local configuration and testing, sourcing representative training data, safe deployment), there is a need to work together. Cross-agency consistency in pre-procurement, ethical assessment, and approval processes is essential. This is especially so in the context of the delegation powers for microdata access under the Data and Statistics Act (where the Government Statistician can give access to microdata to other agencies) and potential uses of AI over microdata in different environments.

There is value in sharing case studies of where things have gone wrong elsewhere to bring the potential risks to life. For example, Horizon - Postmaster saga[2], Robodebt[3], Netherlands Child Welfare[4].

Pre-procurement: Analysis and evaluation phase. Do we need it? Does it need to be AI?

Only procure AI when it will provide the best solution to the problem, and not just the use of ‘shiny new technology’.
Include appropriate evaluation criteria as part of procurement due diligence processes to understand potential risks e.g., privacy, security, bias, transparency, explainability, etc.
Assess that the AI is fit for New Zealand purposes, including such things as using the correct language for those who are disabled.

During the procurement process: Require prospective vendors to communicate what they are doing to manage AI risks.

Compare providers to identify how they contribute to such things as equity, benefits, and relevance to New Zealand systems. It is important to foster local innovation and innovations that don't underserve New Zealanders by using overseas models and algorithms.
Consider if there are options to collaborate with private entities and share benefits.
Ensure that both the procurers and users of AI understand their responsibilities. Not all responsibility lies with the supplier.
Require the completion of privacy and AI impact assessments by both the AI entity and the procuring agency.
Understand the governance practices for these commercial entities in the use of AI and in managing bias, transparency, etc.
Be aware of the risk with closed-source AI e.g., in terms of vendor lock- in.

Post-procurement and governance: Implement and monitor ethical responses within the agency and with the vendor.

Continual monitoring and testing are needed to ensure that the model does not drift, and that bias is not present.
Make Audit Principles explicit upfront to set clear expectations for the public (note that the Global Partnership on Artificial Intelligence (GPAI) is currently developing Audit requirements).
Understand the appropriate place for a human-in-the-loop.
Provide training for those who will use the AI.

Appendix: Additional Guidance for Specific Situations

Additional steps to support informed consent when developing a survey

Create a clear Information Sheet with important information, rather than embedding this in the survey itself where it could easily get lost.
Add in some Consent Statements at the start of the survey that convey exactly what participants are clicking "Agree" to....i.e. that they have read and understood the Information Sheet, had the opportunity to ask questions, know that they can withdraw, and – in cases of data matching – agree to this (e.g., that their responses will be linked to a limited set of variables for the purposes of this study only).
Identify the parties involved in collecting, processing and managing the data, including branding logos.
Identify the project leader or contact person by name and provide fulsome contact details, not a generic email. Whom should participants contact if they have questions/ complaints, or wish to withdraw?
Consider a contact for Māori Cultural Support and any other counselling/ support that may be helpful, depending on the nature of questions being asked.
Explain how the individual's contact address has been obtained.
Detail the anticipated benefits, explaining who will be interested in the results. Some history of the survey and the gains that have resulted would provide valuable context.
Insert a withdrawal statement, providing an indicative date up to which it is possible to withdraw data identifiable as belonging to a person, even if they have completed and submitted the survey.
Explain any consultation on the data collection/survey approach.
Provide reassurance where possible e.g., when data will be deleted, who will have access to it, if it will be deidentified, what it will and won’t be used for, and that it won’t be used for any other purpose.
Distinguish the management of identifiable and coded/ anonymised data (access, storage, destruction).
Consider using a diagram/flow chart for those who are more visual, to show what happens to their data e.g., the data sources, the flow of data, any sharing data with other parties, and the various gates data moves though for linking, de-identifying, etc. over the study lifecycle.