Discussion

Datastore Search is launched!

IATI Technical Team
IATI Technical Team • 8 March 2022

About Datastore Search

 

Datastore Search replaces IATI’s previous Query Builder tool and can be accessed at the same URL: iatidatastore.iatistandard.org/. The new tool offers a range of additional functions to enable complex queries of data published to the IATI Standard:

 

  • Robust free text searches: users can use an online web-search style interface to search for IATI activities using keywords

 

  • Flexible searching: users can select data across any combination of elements or attributes in the IATI Activity Standard

 

  • Extended access: Datastore Search is not paginated so users can download thousands of rows of data at once

 

  • Access of element definitions: Users can hover over any element of the IATI Standard and access its definition

 

Today’s launch follows the release of the Datastore version 3 API in January, which enables organisations to import IATI data into their internal technical systems and external tools. By building Datastore Search on the Datastore version 3, the tool is able to return data that is updated in less than 24 hours and is valid (only data that aligns to the schema of the IATI Standard is provided by the Datastore). 

 

Join a demonstration of Datastore Search tool 22 March, 3pm UTC

 

Find out more about the features of Datastore Search by joining a meeting of IATI’s Data Use Working Group on Tuesday 22 March at 3pm UTC. IATI’s Technical Team will provide a demonstration of the tool and how to use it to access complex data sets to inform your work. Please keep an eye on the DUWG group on Connect for joining instructions: https://iaticonnect.org/DUWG/stream

A recording of the demonstration will be posted online after the meeting. 



 

Feedback and additional Support

 

The first release of Datastore Search offers the ability to filter IATI data by all elements and attributes within the IATI Standard. Users are encouraged to contact the IATI Technical Team with suggestions for additional features and guidance. To gain support to use Datastore Search or for any other data use queries contact the IATI Technical Teaml: support@iatistandard.org. Alternatively, visit the Data Use Query Corner on IATI Connect to learn how to solve data queries with IATI data and post questions to the IATI community.

 

Upcoming work

 

By launching Datastore Search, IATI is helping to meet the needs of data users who have existing knowledge of the IATI Standard and want to download raw IATI data for analysis. This year, IATI’s Secretariat will focus on developing a new version of IATI’s data visualisation tool, d-portal, to make IATI data more accessible and useful to less technical and new audiences. In identifying the requirements for d-portal version 2, including how data is processed and visualised on the tool, the Secretariat will ensure that it complements and not duplicates the functions of Datastore Search.

 

Stay updated with the Technical Team’s work by engaging with IATI’s Technical Community of Practice on IATI Connect.

Comments (14)

leo stolk
leo stolk

Thank you team and congratulations.
Can IATI please tweet and post this news as news on IATISTANDARD.ORG as it is news that should be shared proudly beyond IATI connect. 

theo.sande  

 

Rohini Simbodyal - IATI Secretariat
Steven Flower
Steven Flower

Thanks

> Stay updated with the Technical Team’s work by engaging with IATI’s Technical Community of Practice on IATI Connect.

Please can you point me towards where this Datastore Search project is discussed in the CoP?  I can't find anything, but may not be looking in the right place

It would also be helpful to understand the roadmap for this and other products, and how best we can feed in. 

> Users are encouraged to contact the IATI Technical Team with suggestions for additional features and guidance. To gain support to use Datastore Search or for any other data use queries contact the IATI Technical Teaml: support@iatistandard.org.

As evidenced elsewhere (over the years) I am personally not a fan of pushing such conversations into private emails.  Surely if we want to use and learn with such tools, we should take advantage of public fora.  Is there a commitment to feed back (perhaps to the aforementioned CoP) the common queries and requests from people?

It's great that d-portal is mentioned here.  Just today, I was (with the fantastic support of shi) looking at a complex query whilst using the dquery backend.  It'd be great to understand how such searches could be met in this new tool, and how users can contribute their (increasingly complex) needs.

 

 

matmaxgeds
matmaxgeds

I really like the simple search box, and also the detailed interface exposing all the elements - good envelope pushing, but the simplicity seems to have come at a high cost e.g. the search box does not return what many users will expect, and the detailed interface means you need to have a second screen open to read the standard to know what you want. The user story previously targetted by the QB doesn't seem served by either of the replacements........can the user story of the new interfaces be shared?

As an example, I tried to use it to get projects in 'Sudan' today, the simple interface also returns 'south sudan' projects, so no use there.  So then I have to a) go to 'advanced', b) have to guess the right xml element (just about OK for me), c) scroll for the country code (from a list I have to google EDIT: dropdowns are available - thanks Amy!)).....this was previously 3 seconds with a dropdown in the old QB, which also performed a really good guiding role to likely important filters.

I also searched for FCDO in the search box.....I don't think it returned any FCDO published projects, just ones that mentioned FCDO in the narratives - is that really the expected behaviour?

I guess you have exposed that cool and simple interfaces need a fair amount of smarts behind them to make sense e.g. hints 'were you looking for country/publisher X', and e.g. when selecting an element in the advanced interface, showing the applicable vocabs with narratives (EDIT: solved).

RE D-portal v2 vs datastore v3.....presumably the single search box will be moved to D-portal v2 (which specialises in non-technical users).....and the advanced interface remains as the way for non-api users to generate complex queries?

It would also be nice if the query url was generated e.g. to save for later attempts, is that possible? Also if the links in the top right to advanced/simple were consistent between the two pages i.e. always showing both options.

Mark Brough
Mark Brough

Thanks for sharing this - it's clear that a lot of hard work has gone into this, and it's a technically interesting approach, so congratulations to the team.

It would be great to hear more about who the intended users are for this. I read here that it's been designed with journalists in mind, but it's unclear to me how they would use the search functionality. I also struggle to see how the output formats could be used by journalists without very extensive technical skills.

In recent DUWGs, we discussed three broad sets of users that Datastores have tended to cater to over the years:

(1) bulk export / download, e.g. for powering country systems such as AIMS
(2) aggregation / summary, e.g. for powering simple visualisations or dashboards
(3) analysts, e.g. for running analysis using something like pivot tables in Excel

Of course, this is not a complete typology, but perhaps it's useful for thinking through the objectives of these kinds of tools.

Look forward to discussing further and gathering feedback from everyone at the next DUWG. I'd also echo others' comments, that it would be great to understand more about plans for consultation and development going forward.

Siem Vaessen
Siem Vaessen

Hi everyone,

Not sure why another version of Datastore has been built as https://IATI.cloud does all of the above + so much more. It also allows user to search on https://aida.tools/ where actual use cases for search have been developed with a very clearly defined output into a set of categories (including proper downloads etc.)

Without that it looks as 'Datastore Search' is just an input form in HTML interfacing with Apache Solr w/o any proper specs as to what is expected to be shown to user. Disappointing but probably as expected. Looks like it was delivered like 'hey this queries the Solr index, great!' w/o any consultation with the community / CPO what have you. Good luck with that.

As this 'new' Datastore has not been built according to the actual specifications Datastore V2 (https://IATI.cloud ) was mandated to build against it feels like this attempt at a build for just another Datastore w/o proper specs and looks clunky by the time it was delivered into the field. Like an old T-72

Sorry to say this all looks very IATI 2014 and not IATI 2022....

leo stolk
leo stolk

Dear Siem Vaessen , Siem, to be honest and frank, the reasons to have 'another' Datastore have already been clarified.  IATI invites and encourages the community to use the IATI datastore's API and the current front end. All suggestions for improvement and additional or altered functionality are welcome. Important as these can also inform the requirements of a new version of d-portal, the secretariate will be developing in the coming year.  Leo

Siem Vaessen
Siem Vaessen

Hi leo stolk it's never been clear against what specifications the new Datastore was developed and in all honesty the Secretariat has been pretty vague about it underlying requirements and basically V3 was developed in silo.

Just scanning the original ToR from 2018 that served as the package of requirements for vendors it seems V3 of the Datastore is missing a bunch of features that somehow conveniently were not taken into account for V3 hence that V3 having less functionality than V2 (USD conversion usd being one feature not included).

This ties back into my earlier point on fair competition: how can an initiative like IATI that pretends to uphold all sorts of principles on transparency and accountability not conclude the process for developing V3 was simply unfair and very arbitrary as it did not follow the original ToR. Frankly not really fair play.

 

Steven Flower
Steven Flower

We (Open Data Services Cooperative & 360Giving) first launched a search tool on top of 360Giving data (called GrantNav) in 2016. 

As the initiative has grown, alongside the complexity of data needs of the community, there have been several improvements, including a bunch in late 2021.

All these were coupled with extensive and ongoing user research and testing - including efforts to support those that really wanted to just use freetext search (which can be very hard to get what you need)

GrantNav also illustrates another principle that was discovered: search and then filter (or facet).  Hence, people make a broad search, and then use the available data points to refine down their results.  People also stressed how important it was to download the data in a common and useful format.

It'd be great to see some evidence of this in efforts from IATI. The 360Giving team would be very happy to help,I'm sure.

 

 

Steven Flower
Steven Flower

leo stolk Marion from 360 has kindly offered to run a "fireside chat" at the upcoming Virtual Community Event on the 28th June, which I hope will start some of this important sharing and collaboration

Amy Silcock
Amy Silcock

Thank you all for your feedback so far on Datastore Search. Our aim was to develop and launch the MVP, as per the IATI Governing Board’s steer. We are now gathering comments and feedback to be considered for development as we plan a roadmap for the tool. As matmaxgeds  has mentioned the tool offers a larger scope than previous IATI query builders but does need refinement and further features to help users utilise that scope.

As for collecting feedback, the DUWG are doing a great job of listing out features that different user groups would require. Steven Flower  happy to do similar in the technical CoP. The IATI tech team can present a summary of feature requests already collected as a basis. Aware that a large number of Datastore users aren’t on IATI Connect so parallel collection mechanisms are good. Steven Flower  if the 360Giving team are happy to share their findings on best ways to implement free-text search, and your method for returning results by relevance it would be great to learn from them.

 


Please log in or sign up to comment.