Azure Function and custom extension for Data Lake - it really works!

It seems that this particular topic(extending Azure Functions with custom bindings) has gone very we

It seems that this particular topic(extending Azure Functions with custom bindings) has gone very well(especially taking into account recent news regarding Function, which you can find here). In the final post about Data Lake extension I'll show you two things - how to register your app in AD so you can easily obtain clientId and clientSecret for authentication and how to actually run Data Lake bindings inside Azure Functions runtime!

Registering an application

This is a fairly easy step. You have to:

  • go to Azure Portal
  • find Azure Active Directory blade and then go to App Registrations
  • next click on New application registration - you'll see a form, which you can fill like this:

  • once an application is created go to Keys blade and create a new one - it's your secret
  • clientId in our extension is a value of ApplicationId field from the overview screen

Making it work with Functions runtime

We're prepared to make our extension work with WebJobs runtime, which is a bit different than the one used for Functions. What we do not control when working with Functions are extension, which are registered inside a host. If you try to reference current assembly and run your program, you'll probably see similar screen to this one:

This clearly shows, that there's a problem in discovering our custom binding. Fortunately it's something super easy to fix - as long as you know where the problem occurs. 

Consider following code:

/
public class DataLakeExtensionConfigProvider : IExtensionConfigProvider
{
	public void Initialize(ExtensionConfigContext context)
	{
		var rule = context.AddBindingRule<DataLakeAttribute>();
		rule.BindToInput(attribute => new DataLakeProvider(attribute.ClientId, attribute.ClientSecret));
	}
}

This is an extension config provider, which is needed to allow discovering a binding inside Functions runtime. You may ask what we're doing here? Well, it's pretty straightfoward - we're creating a new binding rule, which allows to bind [DataLake] attribute to DataLakeProvider class responsible for performing all actions. In fact this provider is similar in its purposes to DataLakeAttributeBindingProvider needed for WebJobs.

Additionally we have to show, that our attribute is actually a binding. This requires decorating it with [Binding] attribute:

/
[AttributeUsage(AttributeTargets.Parameter)]
[Binding]
public sealed class DataLakeAttribute : Attribute
{
	public DataLakeAttribute(string clientId, string clientSecret)
	{
		ClientId = clientId;
		ClientSecret = clientSecret;
	}

	public string ClientId { get; private set; }

	public string ClientSecret { get; private set; }
}

If you're missing this attribute, make sure you're using Microsoft.Azure.WebJobs assembly in version 2.1.0-beta1 at least.

Once we have this code we can go a step further - let's create a function, which will really test our extension!

Running a custom extension

To perform actions from this step you'll need VS2017 with Azure Functions SDK installed. If you're ready just add a new function with a timer trigger.

/
public static class DataLakeExample
{
	[FunctionName("DataLakeExample")]
	public static void Run([TimerTrigger("0 */5 * * * *")]TimerInfo myTimer, TraceWriter log)
	{
		log.Info($"C# Timer trigger function executed at: {DateTime.Now}");
	}
}

You'll also need a reference to an assembly containing a custom binding. Now you can extend your function with the logic from the previous examples:

/
public static class DataLakeExample
{
	[FunctionName("DataLakeExample")]
	public static async Task Run([TimerTrigger("*/15 * * * * *")] TimerInfo myTimer,
		[DataLake("clientId", "clientSecret")]
		DataLakeProvider dataLake, TraceWriter log)
	{
		log.Info($"C# Timer trigger function executed at: {DateTime.Now}");

		using (dataLake)
		{
			var path = Path.Combine("This", "Is", "Just", "A", "Test2");
			await dataLake.CreateDirectory(path);
			await dataLake.AppendToFile(Path.Combine(path, "foo"), "THIS IS JUST A TEST");
		}
	}
}

Let's run our function:

What is even more important, data is available inside a Data Lake storage:

Add comment