Demystifying things - WASB in Azure

When you want to interact with Azure Storage, the easiest way is to access it via HTTP client(whether we're talking about the REST API or hitting an endpoint manually). This is fairly easy to detect, consider following URL:

/
https://yourstorageaccount.blob.core.windows.net/path

Simple as that. However, it's possible to find other protocols in the documentation when it comes to using Blob Storage:

/
wasb[s]://<containername>@<accountname>.blob.core.windows.net/<path>

What is this mysterious WASB here and what it is used for?

Hadoop-in-the-cloud

When working with Hadoop, one of its core components is HDFS, which is a file system used by it to manage data and storage. The limitation of HDFS is the fact, that it has access only to local files available for your cluster. There's a question - what if I'd like to access files stored inside by blob storages? Well, this is where WASB comes to play.

WASB - or Windows Azure Storage Blob - is an abstraction built atop of HDFS. It allows Hadoop(or HDIsight because we're talking about Hadoop in the Azure cloud) to seamlessly integrate with Azure Blob Storage. What is more, it allows multiple clusters to access data stored in one place. But what it really gives you?

Sharing is fun!

Before Hadoop can start working on the data, it actually has to load it from somewhere. Normally you either store it in your cluster or load it from an external source. The important thing here is following statement - data has to be accessible locally. Now imagine situation that you'd like to destroy a cluster each time computation has been made(e.g. it happens twice a week and there's no need to pay for it for each day). Moving and loading data each time you want to do something with it consumes time and resources.

When using HDInsight you no longer have to be worried about those caveats. Thanks to WASB, Hadoop can load data from blob storages immediately - you can connect multiple consumers and make computations at the same time. What is more, WASB can be installed with a traditional installations of Hadoop, so even when you provision a cluster on your own in Azure, it's still possible to use WASB in it.