Proposed update to IATI Datastore

This is a request for feedback based on a ticket raised with IATI Support. The issue described below affects the IATI Datastore and Validator (referred to as “the system”). It does not affect other tools such as d-portal or the Country Finance Development Tool.

We think this post will be of interest to anyone that uses the IATI Datastore and Validator regularly, either through the web interface or the API

Leo Stolk from Oxfam Novib has contacted the IATI Secretariat through support@iatistandard.org, because their organisation’s data is unavailable in the IATI Datastore for several hours after they republish their IATI activity files.

One reason why this is happening is that Oxfam Novib’s files are taking a long time to be processed. We are investigating this with Leo, who has been helping us to diagnose the problem.

This issue has highlighted what we believe to be a flaw in the process’s schedule.

Currently, the IATI Datastore update process works as follows:

1. An updated data file is detected
2. Old data is removed from the system
3. The amended data is processed by the system
4. The amended data is loaded onto the system

We propose changing the update process to work in this way:

1. An updated data file is detected
2. The amended data file is processed by the system
3. Old data is removed from the system
4. The amended data file is loaded onto the system

This change means that old data is deleted immediately before loading the new data. If it takes a long time to process the data (Step 2), then the time when no data appears in the system will be minimal.

One consequence of this change is that when data changes, old data will continue to be visible in the system for a bit longer. 

Note that the system is currently set up so that if a file is deleted, then the data is removed as soon as the system detects this. This would not change.

While this is a relatively small change, we want to understand if there might be any unintended consequences that we should be aware of.

Please post your thoughts or questions below by Friday 18 August 2023. If no objections are provided by then, we will implement this change during the week beginning 21 August.

Simon Whitehouse, 

Open Data Services

Open Data Services is the IATI Secretariat’s technology delivery partner. Read more information about IATI’s new Secretariat arrangements.