TLDR: We've updated beta.data.govt.nz, added some more data (old and new, records and hosted data) for testing, and need your feedback to help us decide whether to proceed and what to prioritise.
It has been a couple of months since we released a beta for data.govt.nz and we've had some constructive feedback from both inside and outside of government, so thank you everyone!
We've made some useful updates to the functionality of the beta and have migrated all the working data and records from data.govt.nz for further testing and feedback. We need to determine whether CKAN is the right platform for data.govt.nz and would greatly appreciate your feedback on this point. Apologies for the long blog post, but there is a lot to consider!
We found some of the data on data.govt.nz was out of date so it was good to clean it up and we have now have a tool on beta to help data publishers keep track of any broken links or data, which will improve reliability in future. We've also migrated all data requests into the new data request system, and use cases into the new showcase.
The beta platform is built on CKAN and we have added a number of plugins to test with data publishers and users.
We are also building a roadmap for data.govt.nz which will be blogged in the coming month or so for feedback, after we've had your feedback on the beta site.
In updating beta, we have identified that most agencies do not have the capacity to implement their own data catalog or hosting infrastructure. As such, data.govt.nz can now either harvest dataset metadata records from agencies with catalogs (CSW, CKAN, etc), or it can be used as the metadata catalog and hosting site for agencies that don't have those capabilities. The idea would be that data.govt.nz could be iterated over time to response to the changing needs of data publishers and data users.
Beta.data.govt.nz 2.1 includes the following functionality:
We are working towards being a whole of government data catalog with a standardised metadata schema (based on DCAT and mapped to other schemas such as ANZLIC). Whilst it will take a little more time to get a more complete list of data across government, we are working closely with data publishers to ensure all government data is searchable through data.govt.nz.
If you find government data that isn't on data.govt.nz please let us know and encourage data custodians to list their data on data.govt.nz
We are testing harvesting functionality to automatically populate and regularly update the data.govt.nz catalog with datasets from existing catalogues across New Zealand.
We can easily harvest from CSW (spatial), data.json (inc ARCGIS) and CKAN catalogs and we have developed an easy-to-use spreadsheet for agencies which don't have catalogues to populate data.govt.nz.
Tabular data can be hosted on beta.data.govt.nz by agencies that need somewhere to publish their data assets. If the data is machine readable, an API will be automatically generated, making the data more useful to all data users including the original data publisher!
We are building a basic technical quality framework to automatically grade all datasets according to the most basic needs of data users. We have started with the TBL 5 star quality plugin for CKAN, and have a plan for how to usefully extend this so data users can far more easily identify data they can use.
We have integrated basic CKAN stats and are working on analytics for the site so data users and publishers can look at how often datasets are viewed or downloaded, and we'll work on API calls.
We've mapped the NZGOAL licences to beta.data.govt.nz so data publishers can more easily select an appropriate open license for their open data.
We've added the basic charts plugin to make data that is machine readable easier to play with and gain basic insights without the need for data skills or software. It is just a basic function to help people identify data that may be useful to use, analyse and build further value on.
We have enabled the “Archiver” plugin so we can identify broken links proactively, and to locally cache data files stored on agency websites to make them more accessible. Where such data is machine readable, it will automatically get an API on data.govt.nz and will be browseable as data and through the basic graphs functionality.
Other useful functions on beta you may appreciate include:
The next step is to analyse whether the beta has been successful, whether CKAN is the right platform for data.govt.nz, and if so, for beta to go live to data.govt.nz. As mentioned we are also building a roadmap for data.govt.nz and completing some analysis of the all of government data landscape to identify what is needed to support the broader open government data agenda.
Possible future ideas for the roadmap include:
If you have other ideas, whether you are a data publisher or a data user, please let us know through the beta user survey!
We are working closely with our colleagues across government to collaboratively establish a more data driven public service with data infrastructure you can rely on. We look forward to continuing this journey with you.
Working with Open Data and Information Programme partners, Stats and government agencies, data.govt.nz is making open government data easier to find, publish and use. The new beta.data.govt.nz site released in June 2016 will play an important role in gathering feedback from, and building a better user experience for the open data community.
The data.govt.nz team are working with international counterparts throughout this beta phase, getting further insight into both the platform and best practices with open data release.