Simpro Webhook triggers - seem to get stuck at certain times

[davetar]

Hello Sindre,

Thanks for getting back to us. It was the same yesterday - delays have now climbed to 58 minutes as I write. This problem has been occurring for quite some time, do you have an idea when it may be resolved for good, i.e. when you may add more capacity?

Also - I am glad you are monitoring, but the dashboard is never updated to reflect these API issues. Do you have any plans to include API status on the dashboard?

Thanks

Dave

[Sindre]

Hi Davetar,
You are right, the status dashboard for Simpro does not include API or webhook queues.

There are no current plans to update the dashboard to include these but I will forward the suggestion.

In regards to capacity for our webhook queues I can confirm that we are investigating options to improve this to avoid the lengthy queues that we have seen lately.

Regards

[davetar]

Good morning Sindre

I thought this may help with your monitoring.

Today the delays started at 7:41am East Coast Australia time - within 15 minutes we are already 6 minutes behind and rising. Yesterday delays peaked at over 2 hours. Even at 2am we had 6 minute delays. The day before delays peaked at over 3 hours, and never got back on track throughout the 24 hours.

During business hours we have been 'on time' for a total of 45 minutes since Monday.

While writing this not a single webhook has been sent (over 7 minutes and counting), so the next delay stat I get will be over 10 minutes.

Hope this helps

[davetar]

Sindre

8:30 - we are back on time - no delays!

[jonohowell]

Hi Sindre

As you mentioned the delays are due to high traffic - can you advise what is the long-term strategy to deal with this. As it stands its not a reliable means of notification or event triggering. Are you guys seeking a solution to this problem ? Companies like podium are offering integration into SIMPRO primary based on these webhook triggers. When delays climb like this at random times I can't see the value in those type of integrations. Please can we get an official statement about this issue and a road map plan to address it long-term.

[davetar]

Hello Sindre

More feedback - at 9:31 the delays started again - over the last hour we have received 5 mins worth of webhooks- the current delay has grown to 54 mins over 60 mins. This suggests that some process in simPRO generated an hours worth of webhook send capacity. Probably more as the delays continue to climb.

To be clear 180 webhooks were sent to us in the hour mentioned - we usually get up to 30 twice a minute. And I am sure these are waiting to be sent to us.

I am sure you are aware about what occurred at 9:31 and which build/builds generated 10s of thousands of webhooks to cause this backlog.

As I wrote above - since Tuesday morning this has been an almost constant problem.

Delay is now 59.77 mins as I send..

Thanks

Dave

[davetar]

Update

Delays topped out at 80 mins at 11:16. By 12:17 delays had been cut to 70 mins, and 60 mins by 12:34.

Then it began to go the wrong way - at 13:00 delays are now 72mins

So at about 11:34 something happened on a build/builds to generate lots of webhooks in a very short amount of time.

All of this is based on the assumption that webhooks are sent oldest first

[AidenTew]

Hi davetar,

We appreciate your updates. Rest assured, we're closely monitoring and investigating the issue.