PowerApps

I’ve done a series of blog posts on how to get more than 5000 items from a SharePoint list using Power Automate. While helping someone else out I figured out a way to make it much easier.

If you’re not interested in knowing how it works, just download the flow and reconfigure it for your environment.

Overview

In this example the flow is triggered by PowerApps, but the same method could be used for any trigger. Let’s take a look at the complete flow and then I will explain each step in detail:

Step by Step Explanation

Here is a explanation of every step of the flow in the order they appear in the screenshot above

1. Trigger Action
   Is the flow trigger action. This could be PowerApps, the HTTP connector or anything that suits your situation.
2. Compose Action ‘querySettings’
   This is a compose action which specifies the query settings for the flow.
   
   It has two properties:
   • listName – Specifies the name of the SharePoint list that you want to retrieve items from.
   • fields – Is a comma separated list of fields that will be retrieved from the SharePoint list.
3. Send an HTTP Request to SharePoint ‘getLowestID’
   This step gets the ID of the first record in the SharePoint List.
4. Send an HTTP Request to SharePoint ‘getHighestID’
   Gets the ID of the last record in the SharePoint List.
5. Compose Action ‘threads’
   This flow is going to execute multiple simultaneous queries to SharePoint. Each of which will return a maximum of 5,000 records. Specify the number of threads you will need to cover the number of records that you have in total.
   For example, if you have 9,000 records you will need 2 threads. If you have 23,000 you will need 5 threads. For example:
   [ 0, 1, 2, 3, 4 ]
6. Compose Action ‘itemsPerThread’
   This action calculates the number of items to fetch in each thread. If you have 9,000 items it will get 4,500 items per thread. The expression is:
   add(div(sub(body('getHighestID')?['value'][0]['ID'],body('getLowestID')?['value'][0]['ID']), length(variables('threads'))),1 )
7. Apply to each
   The apply to each action is where the SharePoint queries take place. It is important to configure the concurrency options on this step to ensure it is set to at least the number of threads you have configured:
   
   Within the apply to each the following actions take place:
   1. Compose action ‘lowID’
      Calculates the lowest ID to find in the current iteration:
      add(body('getLowestID')?['value'][0]['ID'], mul(outputs('itemsPerThread'), item()))
   2. Compose action ‘highID’
      Calculates the highest ID to find in the current iteration:
      add(body('getLowestID')?['value'][0]['ID'], mul(outputs('itemsPerThread'), add(item(), 1))
   3. Send an HTTP Request to SharePoint
      This is where the actual SharePoint query takes place. Using the inputs from the previous steps. The expression in the Uri field is:
      _api/Web/Lists/GetByTitle('@{outputs('querySettings')['listName']}')/[email protected]{outputs('querySettings')['fields']}&$filter=ID ge @{outputs('lowID')} and ID le @{outputs('highID')}&$top=5000
      
8. Compose Action ‘Sample Output’
   This action isn’t required for the flow to run. It is used to output two records from the first iteration of the SharePoint API query. This can be used to generate the required JSON Schema in the final step.
   take(outputs('querySharepoint')[0]['body']['value'],2)
9. Response Action
   This is used to send the data back to PowerApps. You need to modify the Response Body JSON Schema. If you run the flow once you can use the output of the “Sample Data” step and use the “Generate from Sample” function.
   
   The body content unions all of the data that is returned from SharePoint into a single array. This will need to be adjusted according to the number of threads that you have configured:

union(
outputs('querySharepoint')[0]['body']['value'],
outputs('querySharepoint')[1]['body']['value'],
outputs('querySharepoint')[2]['body']['value'],
outputs('querySharepoint')[3]['body']['value']
)

Conclusion

This is an incredibly easy way to get more than 5000 items from a SharePoint list. It is also very fast. I was able to get 17,000 rows in 5 seconds and someone I am working with was able to get 5,000 rows in two seconds!

Download the Flow

This flow is simple enough that you don’t really need to understand how it works, you can just import it into your environment and reconfigure to your needs.

Download the flow here and import it into your environment.

This post covers how to configure a Power Automate Flow to get items from a SharePoint list at maximum speed. Super useful if you have an interactive process and you do not want to keep users waiting.

This post is part of a three part series of blog posts, before proceeding please read this post which describes the objectives, pros and cons of each method. It also provides links to the exported flows and sample data which I used to build the flows:

The many ways to retrieve more than 5,000 items from a SharePoint list using Power Automate

If you have not used a parallel branch before, this blog post is worth reading.

Before getting into the technicalities, have a look at the complete flow. If you have already built the more SharePoint API version detailed in this post, it will make sense.

General Premise

This flow retrieves the SharePoint items using the same method as the flow provided in this post. But the work of collecting data from SharePoint is split over four parallel branches, which execute at the same time. Instead of waiting for a single batch of 5,000 to be collected before proceeding to get the next batch.

Changes from the SharePoint API Flow

Several changes were made to the SharePoint API flow:

Variable Declarations

There are some extra values declared and defined at the start of the flow which control how the four branches choose which section of data to return:

Get the Lowest ID from the SharePoint list using the REST API

The lowest ID from the list is retrieved using the HTTP request to SharePoint action and uses the following code:

_api/Web/Lists/GetByTitle('5000 Item Test')/Items?$select=ID&$filter=ID gt 0&$top=1&$orderby=ID asc

Get the Highest ID from SharePoint list using the REST API

Same as the above flow step, but repeated to get the highest ID, uses the following code:

_api/Web/Lists/GetByTitle('5000 Item Test')/Items?$select=ID&$filter=ID gt 0&$top=1&$orderby=ID desc

Gets the Total Item count from the SharePoint List using the REST API

The total item count of the list is returned by using the ItemCount method of the SharePoint API:

_api/Web/Lists/GetByTitle('5000 Item Test')/ItemCount

Declare the control variables for each parallel branch

Because there are four parallel threads, all of the variables which operate within each branch are declared four times.

In the previous example, there was a single integer, array and boolean variable to control the do until loop that was retrieving SharePoint data. Now those variables need to be declared for each parallel branch:

The main control block for the Parallel Branches

In the control block for the parallel branches, a compose action is used to divide the total item count by four to determine the amount of data each branch should retrieve:

div(variables('itemCount'),4)

The control mechanism for the do until loops was modified to be more efficient, by changing the step which checks if there were no results from SharePoint

if(
	empty(body('HTTP_to_SharePoint_to_get_First_Quarter_of_Results').value),true,
	if (
		less(length(body('HTTP_to_SharePoint_to_get_First_Quarter_of_Results').value), 5000),true,false
	)
	
)

This removes an iteration of the loop, by checking if either of the following conditions are true:

• The result set from SharePoint was empty
• The number of results returned from SharePoint was less than 5000

SharePoint API Query for each branch

The SharePoint API query is different for each parallel execution of the data collection.

Each branch collects data according to the batch size defined by dividing the total item count by four and begins at the lowest ID in branch one. ?Next, branch two begins at the end of branch one and so on, so that all items are collected.

Branch 1

_api/Web/Lists/GetByTitle('5000 Item Test')/Items?$select=ID,EmpNo,Title,Last,Furloughed,Telephone,Email,Brand&$filter=ID gt @{variables('intID')} and ID le @{add(body('getLowestID')?['value'][0]['ID'], mul(outputs('DivideTotalByFour'), 1))}&$top=5000"

Branch 2

_api/Web/Lists/GetByTitle('5000 Item Test')/Items?$select=ID,EmpNo,Title,Last,Furloughed,Telephone,Email,Brand&$filter=ID gt @{add(body('getLowestID')?['value'][0]['ID'], mul(outputs('DivideTotalByFour'), 1))}  and ID gt @{variables('intID2')} and ID lt @{add(body('getLowestID')?['value'][0]['ID'], mul(outputs('DivideTotalByFour'), 2))}&$top=5000

Branch 3

_api/Web/Lists/GetByTitle('5000 Item Test')/Items?$select=ID,EmpNo,Title,Last,Furloughed,Telephone,Email,Brand&$filter=ID ge @{add(body('getLowestID')?['value'][0]['ID'], mul(outputs('DivideTotalByFour'), 2))}  and ID gt @{variables('intID3')} and ID lt @{add(body('getLowestID')?['value'][0]['ID'], mul(outputs('DivideTotalByFour'), 3))}&$top=5000

Branch 4

_api/Web/Lists/GetByTitle('5000 Item Test')/Items?$select=ID,EmpNo,Title,Last,Furloughed,Telephone,Email,Brand&$filter=ID ge @{add(body('getLowestID')?['value'][0]['ID'], mul(outputs('DivideTotalByFour'), 2))}  and ID gt @{variables('intID3')} and ID lt @{add(body('getLowestID')?['value'][0]['ID'], mul(outputs('DivideTotalByFour'), 3))}&$top=5000

Compile all results and send back to PowerApps

The response step simply uses a Union action to bring all the SharePoint data back in a single step:

union(variables('arrItems'),variables('arrItems2'),variables('arrItems3'),variables('arrItems4'))

Summary

Using Parallel branches to fetch SharePoint data can provide a huge speed benefit over a standard do until loop at the expense of creating a more complicated flow but it can be worth it to provide the best possible user experience.

It may be easier to just download the exported flow and modify to your requirements than build it from scratch.

You can download the flow and sample data from this post.