Bad Request when inserting data to Azure Table Storage

Recently I faced a bit irritating problem when working with Azure Table Storage - I could not insert an entity because of the following error:

The remote server returned an error: (400) Bad Request.

Yes, this is incredibly helpful message I got there. Let's assume we have following entity schema:

/
private class Entity : TableEntity
{
	public DateTime SomeDateTime { get; set; }
}

So far so good - it couldn't be easier. Now let's use following code:

/
var storageAccount = CloudStorageAccount.Parse("UseDevelopmentStorage=true");
var tableClient = storageAccount.CreateCloudTableClient();
var table = tableClient.GetTableReference("foo");

table.CreateIfNotExists();
table.Execute(
	TableOperation.Insert(new Entity
	{
		PartitionKey = "bar", 
		RowKey = Guid.NewGuid().ToString()
	}));

In the above snippet we are creating a table and immediately inserting an entity. Notice that I didn't set SomeDateTime value - it will be default one. When you execute it, you will get a nice Bad Request error. What did actually happen?

Storage Table types

Sometimes it's easy to forget that Table Storage supports only a subset of data types defined in OData Protocol Specification(because it's possible to query it with OData statements). If you take a look at documentation here, you'll notice following definition:

Edm.DateTime DateTime A 64-bit value expressed as Coordinated Universal Time (UTC). The supported DateTime range begins from 12:00 midnight, January 1, 1601 A.D. (C.E.), UTC. The range ends at December 31, 9999.

So basically the minimum what can be stored in Table Storage is 01.01.1601. Of course DateTime.MinValue is far from correct...

Storage Emulator

What is even more important, there's a difference between handling dates between Table Storage and emulator. Let's add the same rows to both instances:

/
table.Execute(
	TableOperation.Insert(new Entity
	{
		PartitionKey = "bar", 
		RowKey = Guid.NewGuid().ToString(),
		SomeDateTime = new DateTime(1601, 1, 2)
	}));

Here you can find what I got:

  • 0001-01-01T00:00:00.000Z for Emulator
  • 1601-01-01T23:00:00.000Z for Table Storage

Although documentation says:

Date properties in the Table service in the storage emulator support only the range supported by SQL Server 2005 (they are required to be later than January 1, 1753). All dates before January 1, 1753 are changed to this value. The precision of dates is limited to the precision of SQL Server 2005, meaning that dates are precise to 1/300th of a second.

I see that it rather inserts DateTime.MinValue instead what is really confusing.

 

 

Analyzing Table Storage bindings locally in Azure Functions

In one of my current projects I heavily rely on Table Storage bindings used in many of my functions. In fact I have several dozen API functions, which are the base of the whole system. Because codebase grows each day, I needed a tool, which will allow me easily validate whether I query Table Storage correctly - that means I follow some basic principles like:

  • using both PartitionKey and RowKey in a query
  • if RowKey is unavailable - using PartitionKey so I won't have to read the whole table
  • using $top whenever possible so I won't load the whole partition
  • using query projection - leveraging $select for selecting only a subset of columns in a row

In fact I knew two ways of doing that:

  1. Checking logs of Storage Emulator, what I described in this blog post. The disadvantage of that solution is that is logs nearly each and every request so it is hard to find a particular one you're interested in
  2. Using SQL Server Profiler to check what kind of queries are materialized 

As you can see above logs from Storage Emulator are quite detailed, yet painful to work with

I needed a tool, which would combine features of both solutions.

Reading SQL Server Profiler

The idea was to somehow read what SQL Server Profiler outputs when queries are sent to Storage Emulator. Fortunately it is really simple using following classes:

  • SqlConnectionInfo
  • TraceServer

Both are easily accessible in SQL Server directory:

  • C:\Program Files (x86)\Microsoft SQL Server\140\SDK\Assemblies\Microsoft.SqlServer.ConnectionInfo.dll
  • C:\Program Files (x86)\Microsoft SQL Server\140\SDK\Assemblies\Microsoft.SqlServer.ConnectionInfoExtended.dll

There is however a small gotcha. Since SQL Server Profiler is a 32-bit application, you cannot use above classes in 64-bit one. Additionally those assemblies are SQL Server version sensitive - locally I have an instance of SQL Server 2017, if you have other version, you'd have to change the path to point to the correct one.

Does it work?

After some initial testing it seems it works. Let's assume you have following code:

/
[FunctionName("DeviceList")]
public static Task<HttpResponseMessage> DeviceList(
	[HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "device")] HttpRequestMessage req,
	[Table(TableName, Connection = Constants.TableStorageConnectionName)] IQueryable<DeviceEntity> devices,
	[Table(Firmware.Firmware.TableName, Connection = Constants.TableStorageConnectionName)] IQueryable<Firmware.Firmware.FirmwareEntity> firmware,
	[Table(CounterType.CounterType.TableName, Connection = Constants.TableStorageConnectionName)] IQueryable<CounterType.CounterType.CounterTypeEntity> counterTypes,
	[Table(Location.Location.TableName, Connection = Constants.TableStorageConnectionName)] IQueryable<Location.Location.LocationEntity> locations,
	[Identity] UserIdentity identity,
	TraceWriter log)
{
	if (identity.IsAuthenticated() == false) return identity.CreateUnauthorizedResponse();

	var firmwareVersionsCached = firmware.Take(50).ToList();
	var counterTypesCached = counterTypes.Take(50).ToList();
	var locationsCached = locations.Take(50).ToList();

	var query = devices.Where(_ => _.PartitionKey != "device").Take(100).ToList().Select(_ => new
	{
		Id = _.RowKey,
		Name = _.Name,
		SerialNumber = _.SerialNumber,
		Firmware = firmwareVersionsCached.First(f => f.RowKey == _.FirmwareId.ToString()).Version,
		CounterType = counterTypesCached.First(ct => ct.RowKey == _.CounterTypeId.ToString()).Name,
		Location = locationsCached.First(l => l.RowKey == _.LocationId.ToString()).Name
	});

	var response = req.CreateResponse(HttpStatusCode.OK,
		query);

	return Task.FromResult(response);
}

 

Here you can find a part of diagnostic logs from executing above function:

You can find the whole project on GitHub: https://github.com/kamil-mrzyglod/StorageEmulatorTracer. After some time spent with this tools I found planty of issues in my code like:

  • not using PartitionKey
  • reading the same table twice
  • materializing all rows from a table when I needed only a subset

I guess I will even more flaws in the next days.