Over the past few weeks, the IATI Technical Team have been making first steps to build an improved validator. This follows the great work by done by David Carpenter and others who built the initial validator, which has served IATI well since it’s initial development in 2012.
Why a new tool is needed
However, as users call for added validation functionality, the technical team feel would be better served by a revamped version, rather than trying to bolt functionality onto this legacy tool.
The overall aim is to build a new validation tool that features:
- XML syntax and IATI schema checking.
- Content checking, to look for adherance to IATI Rulesets, and codelists. Plus basic sense checking where approproiate (for example: do specified transaction dates fall within the broad range of speficied activity start/end dates).
- RESTful API functionality, that can be used by bespoke publisher applications.
- Improved error messages, showing the context of any errors and pointing users towards appropriate documentation and guidance.
The new tool will also have technical advantages. It will:
- Rationalise the IATI technology stack further towards Python and the ‘lxml’ XML parser.
- Make use of improved software design patterns.
- Enable integration into our other IATI tools, making better use of common functionality.
Our development approach
We are using agile and ‘API first’ approaches to the development of this tool. Work is organised around 2-3 week sprints culminating the the release of a usable and testable product at the end of each iteration.
The improved validator will continue to be an open-source tool and work is taking place on a new branch of the IATI-Public-Validator github repository. As ever, we welcome comments, suggestions and scrutiny on our work.
Sprint 1 complete
The first sprint has just completed and focuses on basic API functionality for XML syntax (well-formed) and validation against an (automatically detected) IATI version.
Examples API usage can be found here with an available public endpoint of: http://dev.validator.iatistandard.org/api/validate/xml
Future sprints will focus on adding documentation, a user-interface and content checking.
Watch this topic for updates.
We will be posting further updates to this thread and deploying to the public instance at regular intervals. In the meantime, we welcome user comments and testing of the current work.
Hi Herman,
You’re right that the current validator seems to struggle with files over 10MB, including the Oxfam Novib dataset that you highlighted, which is around 35MB in size.
It’s unlikely that we’ll work to improve the validator to handle larger inputs, but there’s good news in that it seems that the new validator can handle large files very well. I’ve just used this to confirm that this large file validates against version 2.02 of the schema.
We are working to finalise a user-interface as an alternative to the API too. This has been delayed, partly due to TAG preparation work, but we hope to release the first iteration this in coming weeks.
Hope that this helps in the meantime.