ETL tool
Posted: Wed Mar 22, 2023 9:58 am
Hi,
We need to fetch all records for certain resources but running such queries takes very long and despite retries strategy implemented we often get disconnect errors from the server.
This bulk data is needed for bespoke reports that are not easily available through UI (users would need to click hundreds of records to get all data) or your BI Reporting tool (the underlying tables do not make some of fields connected with records available).
I've read your advice given in the General Discussion on bulk fetching of resources (`display=all` in the query, setting `If-Modicied_Since` in the header)), but also read in your documentation about a possibility of another solution for high volume data extraction:
https://developer.simprogroup.com/apido ... ternatives
Could someone offer some more information about this ETL solution, please? How much is it? What will we get - is it a database dump? What type of data would we get (json, flat files, any other file type)? Will this solution give us all fields connected to records, or is it something like simpro tables in BI Reporting, that make available only limited fields.
We are just trying to choose the best option available for the task and budget for it.
Kind regards,
Grace
We need to fetch all records for certain resources but running such queries takes very long and despite retries strategy implemented we often get disconnect errors from the server.
This bulk data is needed for bespoke reports that are not easily available through UI (users would need to click hundreds of records to get all data) or your BI Reporting tool (the underlying tables do not make some of fields connected with records available).
I've read your advice given in the General Discussion on bulk fetching of resources (`display=all` in the query, setting `If-Modicied_Since` in the header)), but also read in your documentation about a possibility of another solution for high volume data extraction:
https://developer.simprogroup.com/apido ... ternatives
Could someone offer some more information about this ETL solution, please? How much is it? What will we get - is it a database dump? What type of data would we get (json, flat files, any other file type)? Will this solution give us all fields connected to records, or is it something like simpro tables in BI Reporting, that make available only limited fields.
We are just trying to choose the best option available for the task and budget for it.
Kind regards,
Grace