One of Logic Apps biggest selling points is the fact that the runtime is serverless, which results in a very scalable integration platform. I’ve always been amazed by the elasticity characteristics of the Logic App engine, however there’s a huge limitation on the scalability of the out-of-the-box connectors. On many projects, we’ve suffered from this limited connector capacity. This blog post series walks you through the limitations (The bad!), Microsoft’s advised workarounds (The ugly!) and an approach that works great for some connectors (The good!).
This blog series consists of:
- Logic Apps connector performance (1/3) – The bad!
- Logic Apps connector performance (2/3) – The ugly!
- Logic Apps connector performance (3/3) – The good!
Integrations often result in bursty load. Some examples are:
- Huge data export from a line of business application
- Splitting of a big batch file into individual transactions
- More and more event-driven integrations
The integration platform has two important responsibilities. On one hand, it has to process the messages as fast as possible, to meet the business requirements. On the other hand, it sometimes has to slow down (throttle) in case the receiving system can’t keep up with the pace. The latter responsibility can be easily configured via the concurrency control feature. However, on the performance side, the Logic Apps connectors have limitations.
In the connector overview, each connector has documented throttling limits. The throttling happens per API connection. These are the current throttling limits of some important connectors:
|SQL Server||100 calls / 10 seconds||10 calls / second|
|Blob Storage||600 calls / 60 seconds||10 calls / second|
|Event Grid||100 calls / 60 seconds||1,66 calls / second|
|File Storage||600 calls / 60 seconds||10 calls / second|
|SFTP||100 calls / 60 seconds||1,66 calls / second|
|Custom Connector||500 calls / 60 seconds||8,33 calls / second|
Recently, we faced the following business requirement: receive and process 10K messages from an SFTP folder. The business gave us even 1 hour, which was very generous, in my opinion. No big deal we thought! But… for each message, we had at least 3 calls (GET file content, COPY file to archive, DELETE file from folder). This means 30K calls / hour, which equals 8,33 calls / second. Far above the throttling limit of the SFTP connector!
Once a connector starts throttling, it returns an HTTP 429 “Too many requests”. Based on this HTTP status code, Logic Apps will kick off its retry mechanism, which is by default 4 retries with a 20 seconds interval. When there is sporadic throttling, this gracefully resolves the issue. However, when hitting the limits really hard, this results in a cascading effect that makes it only worse.
As a test setup, I’ve created a Logic App that gets fired through an HTTP request. The second action is writing a file to blob storage. By using artillery, I put the Logic App under high concurrent load. This causes throttling and cascading retries, as shown below.
In production-grade integration solutions, it’s not desired that the integration platform is the limiting factor in the end-to-end performance. Typically, the integration platform should be throttled, in order not to overwhelm the backend systems. The Logic Apps connectors should have much higher throttling limits, to fulfil their scalability promise. Several escalations to the product team resulted only in minor improvements. Read in the next parts of this blog post series, how we can work around these limitations.