How to send many requests in parallel in ASP.Net Core

I want to make 1000 requests! How can I make it really fast? Let’s have a look at 4 approaches and compare their speed.

Preparations

In order to test different methods of handling requests, I created a very simple ASP.Net Core API, that return user by his id. It fetches them from plain old MSSQL database.

I deployed it quickly to Azure using App services and it was ready for testing in less than two hours. It’s amazing how quickly a .net core app can be deployed and tested in a real hosting environment. I was also able to debug it remotely and check it’s work in Application Insights.

Herse is my post on how to build an app and deploy it to Azure: http://www.michalbialecki.com/2017/12/21/sending-a-azure-service-bus-message-in-asp-net-core/

And a post about custom data dource in Application Insights: http://www.michalbialecki.com/2017/09/03/custom-data-source-in-application-insights/

API in a swagger looks like this:

So the task here is to write a method, that would call this endpoint and fetch 1000 users by their ids as fast as possible.

I wrapped a single call in a UsersClient class:

 

#1 Let’s use asynchronous programming

Asynchronous programming in C# is very simple, you just use async / await keywords in your methods and magic happens.

Score: 4 minutes 51 seconds

This is because although it is asynchronous programming, it doesn’t mean requests are done in parallel. Asynchronous means requests will not block the main thread, that can go further with the execution. If you look at how requests are executed in time, you will see something like this:

Let’s run requests in parallel

Running in parallel is the key here because you can make many requests and use the same time that one request takes. Code can look like this:

WhenAll is a beautiful creation that waits for tasks with the same type and returns a list of results. A drawback here would be an exception handling because when something goes wrong you will get an AggregatedException with possibly multiple exceptions, but you would not know which task caused it.

Score: 28 seconds

This is way better than before, but it’s not impressive. The thing that slows down the process is thread handling. Executing 1000 requests at the same time will try to create or utilize 1000 threads and managing them is a cost. Timeline looks like this:

Let’s run requests in parallel, but smarter

The idea here is to do parallel requests, but not all at the same time. Let’s do it batches for 100.

Score: 20 seconds

This is the slightly better result because framework needs to handle fewer threads at the same time and therefore it is more effective. You can manipulate the batch size and figure out what is best for you. Timeline looks like this:

The proper solution

The proper solution needs some modifications in the API. It is not effective to fetch users one by one when we need to fetch thousands of them. To further enhance performance we need to create a specific endpoint for our use. In this case – fetching many users at once. Now swagger looks like this:

and code for fetching users:

Notice that endpoint for getting multiple users is a POST. This is because payload we send can be big and might not fit in a query string, so it is a good practice to use POST in such a case.

Code that would fetch users in batches in parallel looks like this:

Score: 0,38 seconds

Yes, less than one second! On a timeline it looks like this:

Comparing to other methods on a chart, it’s not even there:

How to optimize your requests

Have in mind, that every case is different and what works for one service, does not necessarily need to work with the next one. Try different things and approaches, find methods to measure your efforts.

Here are a few tips from me:

  • Remember that the biggest cost is not processor cycles, but rather IO operations. This includes SQL queries, network operations, message handling. Find improvements there.
  • Don’t start with parallel processing in the beginning as it brings complexity. Try to optimize your service by using hashsets or dictionaries instead of lists
  • Use smallest Dtos possible, serialize only those fields you actually use
  • Implement an endpoint suited to your needs
  • Use caching if applicable
  • Try different serializers instead of Json, for example ProfoBuf
  • When it is still not enough… – try different architecture, like push model architecture or maybe actor-model programming, like Microsoft Orleans: http://www.michalbialecki.com/2018/03/05/getting-started-microsoft-orleans/

You can find all code posted here in my github repo: https://github.com/mikuam/Blog.

Optimize and enjoy 🙂

4 thoughts on “How to send many requests in parallel in ASP.Net Core

  1. You may want to check out ServicePointManager.DefaultConnectionLimit, since your requests are to the same point they are most likely only 2 at a time actually running. This is most likely the reason for ~20s to run them, not the thread and context switches because those usually take microseconds. That’s assuming your api can handle many concurrent requests.

  2. “… A drawback here would be an exception handling because when something goes wrong you will get an AggregatedException with possibly multiple exceptions, but you would not know which task caused it…”

    Actually ‘Task.WhenAll()’ won’t return the AggregateException (you can read here: https://stackoverflow.com/questions/12007781/why-doesnt-await-on-task-whenall-throw-an-aggregateexception and here https://codeblog.jonskeet.uk/2011/06/22/eduasync-part-11-more-sophisticated-but-lossy-exception-handling/). Short explanation is that:
    “…Now the team in Microsoft could have decided that really you should catch AggregateException and iterate over all the exceptions contained inside the exception, handling each of them separately. (…) They decided to simply extract the first exception from the AggregateException within a task, and throw that instead…”
    So the output of the ‘…await Task.WhenAll()…” will be basic Exception (first task which failed). The way to handle exceptions from other task can be, eg.:
    “…public static async Task AwaitGracefully(this IList taskList, Action logger)
    {
    try
    {
    await Task.WhenAll(taskList);
    }
    catch (Exception)
    {
    foreach (var task in taskList.Where(t => t.IsFaulted))
    {
    logger(task.Exception?.InnerException ?? task.Exception);
    }
    }
    }
    …”

Leave a Reply

Your email address will not be published. Required fields are marked *