Using the Flow Retry Policy to Resolve 429 - Rate Limit is Exceeded Failures

The Issue

We came upon a situation where an automation we created with Flow was failing due to a rate limit issue.

Specifically, the failure we were seeing was this:

 
429 Rate Limit Exceeded error. Rate limit is exceeded. Try again in 1 second.

429 Rate Limit Exceeded error. Rate limit is exceeded. Try again in 1 second.

 

The Actual Problem

The 429 error is only a symptom of the real issue. Here’s what’s happening in our Flow.

Many Calls to SharePoint / SharePoint Actions

  • The Flow itself consists of a large number of actions and interacts heavily with SharePoint

    • We are doing things like creating libraries, looking up users, setting permissions, creating folders in libraries and items in lists.

    • We are using standard actions for SharePoint and in some cases, we’re using the Send an HTTP Request to SharePoint Action

Many Instances of the Flow Created within a Small Period of Time

  • The Flow is initiated when a new item is added to a SharePoint list.

    • A script loads items into this list

    • In the initial load of the list, a few hundred items may be loaded at once.

      • IMPORTANT

      • We have the ability to reduce / adjust the batch size to reduce the load on that the connector is processing. However, we wanted to see what we could do in Flow first.

Rate Limit Failure

  • The Flow fails occasionally on SharePoint Actions, with the 429 / Rate Limit Exceeded error

    • Below is an example of one of the Actions where the Flow is failing.

 
SharePoint API Action.png
 

About the 429 / Rate Limit Error

Pieter Veenstra wrote this excellent post on another approach to handling the 429 errors generated while using the SharePoint trigger for when an Item is Updated or Modified.

Pieter references the Connectors rate limit in this post. You an see this detail and other important information on the SharePoint Connector’s page.

Connector’s Rate Limit

The chart below (taken from the Connector’s Homepage) shows the rate limit at 600 calls per minute.

 
SharePoint Connector Throttling.PNG
 

Solution! Adjust your Flow Action’s Retry Policy

To reduce the load on the Connector, we chose to adjust the Retry Policy on each action.

Here is a description of the Flow Action’s Retry Policy Setting (taken directly from the Settings on the a SharePoint Action on Apr. 15 2019:

A retry policy applies to intermittent failures, characterized as HTTP status codes 408, 429, and 5xx, in addition to any connectivity exceptions. The default is an exponential interval policy set to retry 4 times.

We already knew that a 429 Error was causing our issue, so this looked like a good place to look for a solution.

You can access the Retry Policy on a SharePoint Action by:

  • Click on the ellipses (…) on the upper right hand corner of the action

  • Then click Settings.

 
Action Settings.png
 

In the Action’s Settings Page, you’ll see that the Retry Policy’s default setting is to use the exponential interval policy to retry 4 times. We chose to modify this to a Fixed Interval Policy.

Here’s how we setup the retry policy:

  • Retry Policy Type: Fixed Interval

  • Retry Count: 10

  • Interval: PT1M (This represents a 1 minute retry interval in the ISO-8601 Data elements and interchange formats standard)

Retry Policy Adjustment.PNG

Results

We found that a Fixed Interval, greater Retry Count and increased Retry Interval was effective in spreading our the load of calls made to the connector.

The Retry Policy adjustment also increased the end to end run time of the Flow. However, we were able to successfully process our Flow runs within an acceptable period of time

Caveats and Considerations

Mileage May Vary

This solution was effective for the load that we were sending to the connector. This may not be the right setup for your Flow. The effectiveness of the Retry Policy is dependent on the load you are sending to the connector.

Consider the following the variables when you are setting up your Retry Policy:

  • What is the volume of calls being made to the connector

  • What is the frequency of the calls being made

Consider your Flow Patterns

Its possible to design Flows in a number of different ways. Consider whether or not an adjustment to your pattern might offer efficiency improvements.

Conclusion

As a final thought, we’re wondering whether an Exponential Interval policy may be even more effective for our use case. Perhaps we’ll investigate this in a subsequent post!

In the meantime, thanks you for reading this! We hope that this post may help you build better Flows.