Appending data to Azure Storage Blob concurrently

Append Blob is a fairly common feature of Azure Storage, which makes all kinds of logs or data aggre

Append Blob is a fairly common feature of Azure Storage, which makes all kinds of logs or data aggregations a piece of cake. While the whole concept is super-easy(at least from the SDK client point of view), using it in a real scenario could give you headaches. Why you may ask? Well, the nature of appends is not so obvious and at the first glance our perceptions could be deceived.

A quick look at the documentation reveals how limited are our options to append data to a blob concurrently

Four Horsemen of the Parallelypse

There're 5 methods in total available on CloudAppendBlob type:

  • AppendBlob
  • AppendFromStream
  • AppendText
  • AppendFromByteArray
  • AppendFromFile

As you can see, I grouped them a little so we have two categories:

  • methods for multiple writers scenario
  • methods for a single writer scenario

Now the question is - how do we know, that one method is designed for this specific scenario? Well, the easiest option is to read the documentation. This is a description of AppendText taken from the API reference:

Appends a string of text to an append blob.
This API should be used strictly in a single writer scenario because the API internally uses the append-offset conditional header to avoid duplicate blocks which does not work in a multiple writer scenario.

So what happens if you try to use such method in a e.g. Azure Function?

The remote server returned an error: (412) The append position condition specified was not met 

This is not the best description, isn't it?

Gimme a code snippet!

The easiest way to fix this issue is to transform a string to a stream:

/
using (var ms = new MemoryStream())
using(var sw = new StreamWriter(ms))
{
	await sw.WriteLineAsync("Serialized_data");
	await sw.FlushAsync();

	ms.Position = 0;

	await blob.AppendBlockAsync(ms);
}

Summary

Personally I found this issue is not so obvious - I involuntarily used AppendText method, which looked as the best match to my code and after some time I noticed those 412 error codes. The one thing you have to remember when using AppendBlock is the fact, that each block cannot exceed 4 MB size each. This - along with the limit of 50k append operations - allows for building a blob of max size equal to 195 GBs, which should be fine for the most of projects.

Add comment