DQI Consultation: Data Completeness - 2.9.1 - 2.9.4 Results / 2.10.1 and 2.10.2 Document Links / 2.2 Location

2.10.1 methodology seems to perhaps be copied incorrectly as it is the same as the humanitarian methodology.  

2.10.2 if the dashboard can deal with multiple activity files, is there a reason it can't deal with multiple organizational files?  I am a tad confused as to why there would be multiple org files but that is an issues, why is it allowed?

We have some general comments on this phase which can be found in the numbered points in the document below :  

https://docs.google.com/document/d/1BpEjcypxpHJ_SNCnscy2HZe7otEZZIYcjE7HPFdDPfI/edit?usp=sharing

As well as some indicator specific comments :

2.9.1 - 2.9.4 Results / 2.10.1 & 2.10.2 Document Links / 2.2 Location 

2.9.1>2.9.4 Results 

• Challenges with results include: heterogeneous approaches to publishing results, and timescales - how old should the activity be to require actuals 

• Results are spread across several IATI elements - which will be tested as results can be documents or data? Many organisations publish results only quarterly or bi-annually. How will publication rate be accounted for? 

• Many qualitative results will not have a baseline measure  

2.10.1 Document links percentage: 

• Seems like an error – assessment description is not related to document links 
   

2. Document links Boolean: 
    

• What about all the quality checks on the document such as; date, content, working URLs, appropriateness etc. Is just one document link valid? This represents a very entry level check on ‘data quality’ 

• This assessment is suggesting to exclude multiple org. files which fall under one publisher ID, but several major organisations do use this approach to publishing their data (the United States, for example). Will these major agencies then be excluded?  

2.2 Location: 

• What about the accuracy of subnational locations? We often find that the given location is inconsistent with the documentation during the Aid Transparency Index sampling or that a non-specific central location may be used (in extreme cases this is off in the ocean somewhere). 

General comment on RESULTS: ILO proposes that the measures related to RESULTS should be applied to activities with “closed” status that ended within the last 12 months (or some other period to be discussed). Many organisations analyse the results achieved after the completion of the activity, so it is logical not to ask publishers to report results for activities that just started or are on-going. Even though the monitoring systems might be in place and might collect some data on the progress towards the achievement of results, these data might be sensitive and difficult to publish in IATI. In our view, publishers should not be punished if they don’t publish results for on-going activities, but they should be held accountable for publishing results of recently completed activities.

2.10.1 Waiting for the correction of the mismatch between the title and the methodology to provide the feedback.

We generally agree with the comments from Emma Jenkins regarding results (2.9.1 - 2.9.4) and document links (2.10.1 - 2.10.2). 

On result, many types of interventions would not be expected to have quantitative types of results, therefore cannot have a baseline or target. Since the IATI Tech Team does not have the resources required to perform detailed assessments but rather must rely on automatic tests of whether *any* information is provided, we recommend keeping only 2.9.1 but applying it to a different group of activities based on dates/status (as suggested by ILO colleague). 

2.10.1: whether there is a document link will say nothing about data completeness or quality. A single document link in an activity could be anything. Is it relevant? Is is accessible? Is it properly coded? The DQI is already too long/complicated; we recommend removing this element.

2.10.2: Similarly, we recommend removing this element from the DQI. 

2.2: the proposed methodology does not align with the IATI Standard. The standard 1) states that the location can be indicated with a gazetteer reference, and 2) should should include the reach (or scope), which makes the location details more meaningful. We cannot exclude gazetteer references from the DQI.

Anna Whitson - IATI Secretariat  , there is a mistake in the methodology paper for this section as well. Could the Tech Team please provide an updated paper? 

Thanks for pointing that out, Yohanna! Tagging Amy Silcock here for her action.

I think many of my comments have been mentioned already but I'll list them anyway.

Results:

General: this is difficult for active activities. Even baselines can easily take a year to collect. Targets the same. Actuals sometimes (especially on impact and outcome) are only known after the activity closes.

2.9.1. It counts number of active activities that have a valid result/indicator element. I'd simplify this and focus on indicators. Results actually don't do much. They are, bluntly said, a group of indicators.

2.9.2. + 2.9.3.

Why only active activities? (which is my feedback on the whole DQI)

2.9.4. Do this on closed activities, I'd say.

What I am missing here, is checks on a lot of other elements of the results section:

- aggregation status
- Result type (which should actually move to indicator and become indicator type but that's a whole other discussion...)
- measure
- vocabulary + code + vocabulary uri

Document links

Genuine question: has anyone here ever used the document links in their analysis of IATI data?

Location

Particularly difficult for sensitive activities

Q1. The US provides one org file (with many documents therein), and agrees with Michele in that we don’t see an apparent valid reason for including multiple org files - have any been given by other publishers?  That said, multiple submissions seems better than none and perhaps shouldn’t be penalized in the same way. 

Other: regarding results, the US would likely agree with only assessing results for closed activities. Here again, activities should be filtered by aid type. There is no reason that government budget support, multilateral contributions, or administrative costs should have results associated with those activities.

In response to a comment on the utility of document links:  Providing documents links should be included in any assessment. IATI is the only data standard that allows a way to publish this type of information. If it is not being used, it’s not because it is not useful, it is because the data is not being made available to data users. 

Thank you for your comments. We'll review the idea of including results within closed activities. If we were to add this in what time period should be included? E.g. activities that were closed less than 5 years ago? Aware that if we include all activities some organisations will be penalised by having activities from 50 years ago which they are not able to provide results for.

For the document link question, 71 organisations provide a document link at results level, 2 at indicator level. Like Sarah's mentioned it's not widely available, accessible or known about.

For Thea Schepers's point on capturing more data in a results element do you have a data use case for why these should captured? Keen to hear others thoughts on if they would be useful to include.

For which subset of organisations different metrics should be captured will come up in the next round. Sarah McDuff - IATI Secretariat   looking at aid-type should be considered here.