Proposed update to IATI Datastore

This is a request for feedback based on a ticket raised with IATI Support. The issue described below affects the IATI Datastore and Validator (referred to as “the system”). It does not affect other tools such as d-portal or the Country Finance Development Tool.

We think this post will be of interest to anyone that uses the IATI Datastore and Validator regularly, either through the web interface or the API

Leo Stolk from Oxfam Novib has contacted the IATI Secretariat through support@iatistandard.org, because their organisation’s data is unavailable in the IATI Datastore for several hours after they republish their IATI activity files.

One reason why this is happening is that Oxfam Novib’s files are taking a long time to be processed. We are investigating this with Leo, who has been helping us to diagnose the problem.

This issue has highlighted what we believe to be a flaw in the process’s schedule.

Currently, the IATI Datastore update process works as follows:

An updated data file is detected
Old data is removed from the system
The amended data is processed by the system
The amended data is loaded onto the system

We propose changing the update process to work in this way:

An updated data file is detected
The amended data file is processed by the system
Old data is removed from the system
The amended data file is loaded onto the system

This change means that old data is deleted immediately before loading the new data. If it takes a long time to process the data (Step 2), then the time when no data appears in the system will be minimal.

One consequence of this change is that when data changes, old data will continue to be visible in the system for a bit longer.

Note that the system is currently set up so that if a file is deleted, then the data is removed as soon as the system detects this. This would not change.

While this is a relatively small change, we want to understand if there might be any unintended consequences that we should be aware of.

Please post your thoughts or questions below by Friday 18 August 2023. If no objections are provided by then, we will implement this change during the week beginning 21 August.

Simon Whitehouse,

Open Data Services

Open Data Services is the IATI Secretariat’s technology delivery partner. Read more information about IATI’s new Secretariat arrangements.

Comments (7)

Michelle Levesque • 4 months ago

A question. Since the old data will not be deleted until the new data has loaded, if someone goes to the datastore at the point that new is going in but old isn't yet deleted, will they get duplicates (old and new) or is there a way to know not to pick up duplicate data?

Is there some sort of notification/warning to the general user of the datastore to know when a refresh is in process and then completed?

James (IATI Secretariat) • 4 months ago

Hello Michelle,

> Since the old data will not be deleted until the new data has loaded

Actually that's not quite right - the proposal is that "Old data is removed from the system (step 3)" and then "The amended data file is loaded onto the system (step 4)".

> will they get duplicates (old and new)

Because of the ordering, it shouldn't happen that duplicates are ever seen. We would keep an eye on that situation and treat it as a bug if it did.

> Is there some sort of notification/warning to the general user of the datastore to know when a refresh is in process

Generally speaking no - the datastore is usually always refreshing something! As soon as it notices data is updated/deleted it processes it. It is checking and running 24/7.

It is possible for someone to dig into the details of a specific publisher. Go to https://validator.iatistandard.org/ , select "Public Data Viewer", and select a publisher. That will tell you the state of the publishers data in the data store. (As part of this work, we would add a little bit more information to this screen.)

Thanks,
James,
IATI Secretariat

Yohanna Loucheur • 4 months ago

The proposed change seems to make eminent sense in minimizing the time when no data is available for a given publisher.

leo stolk • 4 months ago

Thanks Yohanna, basically Oxfam Novib's data is not available during 3-4 hours, during the morning of each day in Europe.

James (IATI Secretariat) • 3 months ago

Thanks for comments. It will take a while till this as released, as we will want to test this carefully on staging servers. We will update with news later.

James (IATI Secretariat) • 1 month ago

This has now been released. Thanks for bearing with us while we tested this carefully.

The pipeline has a changed order of actions - now, when updated data is detected we process the new data before deleting the old data.

(If a file is removed from the registry, we still delete the data as soon as we can in line with data removal: https://iatistandard.org/en/data-removal/ )

We have updated the validator screens to provide more information during processing.

As always, any problems or issues do please let us know via IATI Connect or our support email address.

leo stolk • 1 month ago

Thanks James and colleagues, great that our data is now available throughout the day in the datastore, without interruption.

Please log in or sign up to comment.