ETL tool

Posts about the simPRO API. Help/support/how-to etc.
Forum rules
  • Check the documentation first. Before posting questions, please check the documentation to ensure it doesn't already cover what you need.
  • No spam. All automated messages, advertisements, and links to competitor websites will be deleted immediately.
  • Post in relevant sub-forums only. Messages posted in the wrong topic area will be removed and placed in the correct sub-forum by moderators.
  • Respect other users. No flaming or abusing fellow forum members. Users who continue to post inflammatory, abusive comments will be deleted from the forum after two warnings are issued by moderators.
  • Harassment. No threats or harassment of other users will be tolerated. Any instance of threatening or harassing behavior is grounds for deletion from the forums.
  • Adult content. No profanity or pornography is allowed. Posts containing adult material will be deleted.
  • Bandwidth. All images and signatures must be 500 x 500 pixels or smaller. Posts containing over-sized images and signatures will be removed.
  • Illegal content. No re-posting of copyrighted materials or other illegal content is allowed. Any posts containing illegal content or copyrighted materials will be deleted.
Post Reply
Grace
Posts: 12
Joined: Wed Mar 09, 2022 9:56 am

ETL tool

Post by Grace » Wed Mar 22, 2023 9:58 am

Hi,

We need to fetch all records for certain resources but running such queries takes very long and despite retries strategy implemented we often get disconnect errors from the server.
This bulk data is needed for bespoke reports that are not easily available through UI (users would need to click hundreds of records to get all data) or your BI Reporting tool (the underlying tables do not make some of fields connected with records available).
I've read your advice given in the General Discussion on bulk fetching of resources (`display=all` in the query, setting `If-Modicied_Since` in the header)), but also read in your documentation about a possibility of another solution for high volume data extraction:
https://developer.simprogroup.com/apido ... ternatives

Could someone offer some more information about this ETL solution, please? How much is it? What will we get - is it a database dump? What type of data would we get (json, flat files, any other file type)? Will this solution give us all fields connected to records, or is it something like simpro tables in BI Reporting, that make available only limited fields.

We are just trying to choose the best option available for the task and budget for it.

Kind regards,

Grace
Top
Grace
Posts: 12
Joined: Wed Mar 09, 2022 9:56 am

Re: ETL tool

Post by Grace » Fri Mar 24, 2023 9:18 am

Hi,

As it's unusual on this forum to wait for a reply for longer than a day or two, I guess this query is a hot potato?
If this is not in your team's remit, can you maybe point us in the direction of who we should ask about your "hight volume data extraction" that is mentioned in the API documentation, please?

Thank you

Grace
Top
Sindre
Posts: 493
Joined: Wed Apr 20, 2022 4:51 am
Location: Brisbane

Re: ETL tool

Post by Sindre » Sun Mar 26, 2023 11:34 pm

Hi Grace,
Sorry for not getting back to you last week.

For data extraction you can contact simPRO support who will help you with more information about our daily database extraction options.

Please contact them through this website:
https://helpdesk.simprogroup.com/

Regards
Sindre
API Developer | Simpro Software Group
Top
Post Reply

Return to “General Discussion”