Dapr sidecar Race Conditions in .NET when using Secret Stores?

TL;DR: Although the Dapr sidecar is stating that it blocks untils the application is ready to serve on its port, we can still access Secrets through it and use them in our application startup.

When using Dapr, you might have probably seen this in the sidecar logs.

INFO[0000] application protocol: http. waiting on port 5231.  This will block until the app is listening on that port.
✅  You're up and running! Dapr logs will appear here.

This made me wonder if we could run into a race condition, where the Dapr sidecar is blocked by waiting on the application to start, and the application cannot fully start and serve on that port because it might need to connect to Dapr in the startup process.

This could especially happen, when using ASP.NET in combination with Dapr's Secrets management. Let's say, ASP.NET needs to fetch Secrets from Dapr in the Setup, before it can serve the Controllers and Dapr is waiting for the controllers to come up before serving Secrets as shown in the Program.cs file below.

using Dapr.Client;
using Dapr.Extensions.Configuration;
using DaprRaceConditions.Services;

var builder = WebApplication.CreateBuilder(args);

// Add the secret store Configuration Provider to the configuration builder.
var daprClient = new DaprClientBuilder().Build();
builder.Configuration.AddDaprSecretStore("secret-store", daprClient);

// Add services to the container.
builder.Services.AddSingleton(daprClient);
builder.Services.AddSingleton(new FooService(builder.Configuration["FooSecret"] ?? "No Secret Found"));

builder.Services.AddControllers();
builder.Services.AddEndpointsApiExplorer();

var app = builder.Build();

// Configure the HTTP request pipeline.
app.UseHttpsRedirection();
app.UseAuthorization();
app.MapControllers();
app.Run();

To find out, if the Dapr process is really blocking, let's start it separately without the app.

dapr run --app-id "daprraceconditions" --app-port "5231" --resources-path "./Components"

WARNING: no application command found.

ℹ️  Starting Dapr with id daprraceconditions. HTTP Port: 65530. gRPC Port: 65531
INFO[0000] starting Dapr Runtime -- version 1.9.0 -- commit fdce5f1f1b76012291c888113169aee845f25ef8  app_id=daprraceconditions instance=Robin-Manuels-MacBook-Pro-16.local scope=dapr.runtime type=log ver=1.9.0

...

INFO[0000] component loaded. name: secret-store, type: secretstores.local.file/v1 

...

INFO[0000] application protocol: http. waiting on port 5231.  This will block until the app is listening on that port.  app_id=daprraceconditions instance=Robin-Manuels-MacBook-Pro-16.local scope=dapr.runtime type=log ver=1.9.0
✅  You're up and running! Dapr logs will appear here.

We can also see, is that the Secret Store secret-store itself has been loaded, but it still gives us the "This will block..." message. So let's find out, if the can access the Secret Store via API.

curl http://localhost:50183/v1.0/secrets/secret-store/FooSecret

{"FooSecret":"Bar"}

This worked. So the Dapr Sidecar process itself doesn't really seem to be blocking or waiting at all. Therefore, the Info message is very misleading.

As we now know that the Sidecar itself works perfectly fine without the app actually serving on its app port (in this case 5231), we only need to make sure, that the app does not start and try to fetch a secret before the Dapr sidecar is ready. Especially when running in a container scenario, we need to make sure, that we wait for Dapr Sidecar to be ready. For this, we can add a TimeSpan to the .AddDaprSecretStore(...) method. If the omit that, the default is 5 seconds.

builder.Configuration.AddDaprSecretStore("secret-store", daprClient, TimeSpan.FromSeconds(10));

And that's it. We found out, that the Dapr sidecar is not blocking at all although saying that in the logs and that we can easily wait for the sidecar to be started in the .NET SDK.