Is Event Grid faster than Azure Functions? #1

It's over two months since the last blog post, so it's time for a big come back!

In the very first post in 2018, I'd like to test whether Event Grid brings us improvements when it comes to calling registered subscribers over an old-fashioned Blob Trigger in Azure Functions. I decided to start with this topic mostly because it's no more than a few days since Event Grid's GA was announced. Let's start!

Set-up the architecture

To performs a basic test(in fact I divided this post into two parts), we'll need two things:

  • Function App with 3 functions(publisher, blob trigger and HTTP endpoint)
  • Event Grid instance
  • General purpose Storage Account V2 - link

Why do we need General-purpose v2 (GPv2) account? Well - new Event Grid storage trigger requires updated version of an account, there's nothing we can do about it. The good thing is the fact, that you may upgrade your account to GPv2 using e.g. this command:

/
Set-AzureRmStorageAccount -ResourceGroupName <resource-group> -AccountName <storage-account> -UpgradeToStorageV2

HTTP endpoint and Event Grid subscription

To create a subcription in Event Grid from a storage to a function we have to use of the following methods:

  • Azure CLI
  • Powershell
  • REST API
  • SDK

Unfortunately for now it's not possible to subscribe to storage events using Azure Portal. For the purpose of this test I decided to use Azure CLI in the portal:

/
az eventgrid event-subscription create --resource-id "/subscriptions/55f3dcd4-cac7-43b4-990b-a139d62a1eb2/resourceGroups/kalstest/providers/Microsoft.Storage/storageaccounts/kalsegblob" --name es3     --endpoint https://contoso.azurewebsites.net/api/f1?code=code

You can find the full reference to the command here.

You can easily use Cloud Shell here, which is available within the portal

If you run the command now, you'll be suprised - it's not possible to create a subscription because your endpoint is not authenticated. What the heck you may ask? Well, this is all described in the documentation of Event Grid. To make the long story short - each time you try to add a new endpoint, which will be used to send events to, it has to be validated. TO validate your endpoint, Event Grid sends a message similar to this:

/
[{
  "id": "2d1781af-3a4c-4d7c-bd0c-e34b19da4e66",
  "topic": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "subject": "",
  "data": {
    "validationCode": "512d38b6-c7b8-40c8-89fe-f46f9e9622b6"
  },
  "eventType": "Microsoft.EventGrid.SubscriptionValidationEvent",
  "eventTime": "2018-01-25T22:12:19.4556811Z",
  "metadataVersion": "1",
  "dataVersion": "1"
}]

What you have to do is to respond to such request using validationCode it sent:

/
{
  "validationResponse": "512d38b6-c7b8-40c8-89fe-f46f9e9622b6"
}

How to achieve it in our test? We'll develop our HTTP function and perform quick deserialization just to have our enpoint validated. Once it's done, we can switch function's content with a proper logic:

/
[FunctionName("Http")]
public static async Task<HttpResponseMessage> Run(
	[HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)] HttpRequestMessage req,
	TraceWriter log)
{
	var data = await req.Content.ReadAsStringAsync();
	var @event = JsonConvert.DeserializeObject<Event[]>(data)[0];

	return req.CreateResponse(HttpStatusCode.OK, new { validationResponse = @event.Data.ValidationCode});
}

public class Event
{
	public string Topic { get; set; }
	public string Subject { get; set; }
	public string EventType { get; set; }
	public DateTime EventTime { get; set; }
	public Guid Id { get; set; }

	public ValidationRequest Data { get; set; }
}

public class ValidationRequest
{
	public string ValidationCode { get; set; }
}

Once you publish this function, you can run the command mentioned to register a new event subscription.

Functionality

These are function I used to perform the first part of the test:

PUBLISHER

/
[FunctionName("Publisher")]
public static async Task Run([TimerTrigger("*/10 * * * * *")]TimerInfo myTimer,
	[Blob("functionsgrid/blob", FileAccess.Write, Connection = "FunctionsGrid")] Stream blob,
	TraceWriter log)
{
	log.Info($"C# Timer trigger function executed at: {DateTime.Now}");

	using (var sw = new StreamWriter(blob))
	{
		await sw.WriteAsync(JsonConvert.SerializeObject(new Blob()));
		log.Info("Blob created!");
	}
}

[FunctionName("Publisher2")]
public static void Run2([TimerTrigger("*/10 * * * * *")]TimerInfo myTimer,
	[Blob("functionsgrid/blob2", FileAccess.Write, Connection = "FunctionsGrid")] out string blob,
	TraceWriter log)
{
	log.Info($"C# Timer trigger function 2 executed at: {DateTime.Now}");

	var o = new Blob { Text = File.ReadAllText("file.txt") };
	blob = JsonConvert.SerializeObject(o);
}

public class Blob
{
	public Blob()
	{
		Id = Guid.NewGuid();
		Created = DateTime.Now;
	}

	public Guid Id { get; set; }

	public DateTime Created { get; set; }

	public string Text { get; set; }
}

HTTP

/
[FunctionName("Http")]
[return: Table("Log", Connection = "FunctionsGrid")]

public static async Task<Blob.LogEntity> Run(
	[HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)] HttpRequestMessage req,
	TraceWriter log)
{
	var dateTriggered = DateTime.Now;
	var data = await req.Content.ReadAsStringAsync();

	var @event = JsonConvert.DeserializeObject<Event[]>(data)[0];
	log.Info($"Processing {@event.Id} event.");

	var storageAccount = CloudStorageAccount.Parse("DefaultEndpointsProtocol=https;AccountName=functionsgrid;AccountKey=l52CpYyO4D30m3UoGk/jTruzYo1HuvTlQjvGWTG1wZeN01n4YLK1zwdy6VS6D6tN26YUXzuQcQKXZDdMOr0X9g==;EndpointSuffix=core.windows.net");
	var blobClient = storageAccount.CreateCloudBlobClient();
	var blob = blobClient.GetBlobReferenceFromServer(new Uri(@event.Data.Url));

	using (var sr = new StreamReader(blob.OpenRead()))
	{
		var readToEnd = sr.ReadToEnd();
		log.Info(readToEnd);
		var fileBlob = JsonConvert.DeserializeObject<Publisher.Blob>(readToEnd);
		log.Info("Text: " + fileBlob.Text);
		if (string.IsNullOrEmpty(fileBlob.Text) == false)
		{
			return new Blob.LogEntity("eventgrid_big")
			{
				BlobCreated = fileBlob.Created,
				BlobProcessed = dateTriggered
			};
		}

		return new Blob.LogEntity("eventgrid")
		{
			BlobCreated = fileBlob.Created,
			BlobProcessed = dateTriggered
		};
	}
}

public class Event
{
	public string Topic { get; set; }
	public string Subject { get; set; }
	public string EventType { get; set; }
	public DateTime EventTime { get; set; }
	public Guid Id { get; set; }

	public EventData Data { get; set; }
}

public class EventData
{
	public string Url { get; set; }
}

BLOB

/
[FunctionName("Blob")]
[return: Table("Log", Connection = "FunctionsGrid")]
public static LogEntity Run([BlobTrigger("functionsgrid/{name}", Connection = "FunctionsGrid")]Stream myBlob, string name, TraceWriter log)
{
	var dateTriggered = DateTime.Now;
	log.Info($"C# Blob trigger function Processed blob\n Name:{name} \n Size: {myBlob.Length} Bytes");

	using (var sr = new StreamReader(myBlob))
	{
		var content = sr.ReadToEnd();
		var blob = JsonConvert.DeserializeObject<Publisher.Blob>(content);

		if (blob != null && string.IsNullOrEmpty(blob.Text) == false)
		{
			return new Blob.LogEntity("function_big")
			{
				BlobCreated = blob.Created,
				BlobProcessed = dateTriggered
			};
		}

		return new LogEntity
		{
			BlobCreated = blob.Created,
			BlobProcessed = DateTime.Now
		};
	}
}

Results

All results were saved to a table in Table Storage. I measured the exact time when a function starts its execution - the results don't care about how long a function needed to perform all tasks. There were two scenarios:

  • Upload a file with a simple JSON content, once per 10 seconds
  • Upload a 2.5MB file, once per 10 seconds

Here are the results(for 1653 executions):

What can you say about this chart? I assume these are some initial conclusions:

  • although the difference was a matter of miliseconds, Event Grid seems to notify subscriber almost with no delay while Azure Functions need to poll storage and wait for new files
  • bigger file means more delay when it comes to notification, which is true for both Grid and Functions
  • this was a simple smoke test - when we start to push more files, it's possible that result from today won't be relevant
  • there's one interesting observation - bigger file seems to be processed much slower in HTTP function, which has to download a file after being notified about its existence

In the next episode we'll try to stress this solution a little bit to check how it behaves when it comes to handling many small and bigger files. Stay tuned!

Reactive Durable Functions

Durable Functions itself are a big topic and I'll come back to the them soon. In the previous post I created a simple Function App, which inserts a row into Table Storage. All activities were orchestrated and it was really easy to schedule more work. Today I want to present you how easy you can transform such active architecture into a passive one using Event Grid. This will be a fairly easy episode so let's start!

Function App

There's no need to change anything in the Function App since for now we'll use a HttpTrigger. Unfortunately we cannot host it locally(because of Event Grid) so there's a need to publish it to the cloud.

Event Grid

You can easily deploy Event Grid from the marketplace. For now there's nothing special regarding its installation, so I won't go into details.

Event Grid chosen from Marketplace - still in preview though

Combining it all together

Once we have components deployed we can configure Event Grid so it'll pass events to the chosen endpoint. In our case it'll be our Galaxy_Create_Start function(which we used to start orchestration). We'll need one thing - our function URL. To get it go to the Function App you deployed, find a function and click on Get Function URL.

Once you have it, we can go to Event Grid and create a new subscriber.

What we need now is a new subscription. This feature allow you to orchestrate events flow, so each subscriber can be subscribed to a particular event type. With this configured we can centralize the way, how e.g. multiple services built with Azure Functions integrate with event producers.

My subscriber configured and subscribed to nebula.galaxy event type

When you configure a new subscriber it will be added to the list of all supported subscribers. We've done all what we needed to integrate Event Grid with Azure Functions, let's test it now.

Working example

There're two important things what we need to test our solution - Event Grid endpoint and access key. The former ss available on the main screen - please copy it so you'll know where to post your messages. Access keys can be found under Settings section in Event Grid main menu.

Since we don't have any real producer yet, we'll try to simulate one. For this purposed I used Postman, however all is up to you. The only thing we have to do is to post a HTTP request to the mentioned enpoint. Here you have an example:

/
POST /api/events HTTP/1.1
Host: your.eventgrid.azure.net
aeg-sas-key: YOUR_KEY
Content-Type: application/json
Cache-Control: no-cache

[
    {
        "id": "2",
        "eventType": "nebula.galaxy",
        "subject": "nebula/galaxy/create",
        "eventTime": "2017-11-08T13:25:00+01:00",
        "data":{
        }
    }
]

As you can see the payload has a specific schema, which will be validated on the Event Grid side. In fact it should self-explanatory. What is important here is the fact, that this payload is being passed to the function. If you change the main function a little bit:

/
[FunctionName("Galaxy_Create_Start")]
public static async Task<HttpResponseMessage> StartOrchestration(
	[HttpTrigger(AuthorizationLevel.Function, "post", Route = "orchestration/start")] HttpRequestMessage req,
	[OrchestrationClient] DurableOrchestrationClient starter,
	TraceWriter log)
{
	// Function input comes from the request content.
	string instanceId = await starter.StartNewAsync("Galaxy_Create", null);

	var payload = await req.Content.ReadAsStringAsync();
	log.Info($"Started orchestration with ID = '{instanceId}'.");
	log.Info($"The payload is: {payload}");

	return starter.CreateCheckStatusResponse(req, instanceId);
}

You'll see following result when you go to the function and check the console:

/
2017-11-08T12:36:54.565 Function started (Id=ecb2655e-912e-435b-b916-f21b65729716)
2017-11-08T12:36:55.144 Started orchestration with ID = 'f7d2ad0001204ff0a381d61b448ef8b7'.
2017-11-08T12:36:55.144 The payload is: [{
  "id": "3",
  "eventType": "nebula.galaxy",
  "subject": "nebula/galaxy/create",
  "eventTime": "2017-11-08T12:25:00+00:00",
  "data": {},
  "topic": "/SUBSCRIPTIONS/____________/RESOURCEGROUPS/NEBULA-EUW-DEV-RG2/PROVIDERS/MICROSOFT.EVENTGRID/TOPICS/NEBULA-EUW-DEV-EVENTGRID"
}]
2017-11-08T12:36:55.173 Function completed (Success, Id=ecb2655e-912e-435b-b916-f21b65729716, Duration=596ms)

Of course you can easily deserialize it and incorporate into your flow.

Summary

As you can see implementing Event Grid as a gateway to the underlying architecture is a piece of cake. In fact we didn't need any change in our code - the whole integration perfomed seamlessly. In the next episode I'll try to present you how to integrate a producer(Event Hub), so we don't have to post messages directly to our Event Grid.