Log4Net: writing log entries to Azure Table Storage

August 15, 2011

...

Earlier, I blogged about how one can leverage Azure Diagnostics to easily set up Log4Net for your application running in Azure, and how to customize the log entries for the Azure environment.

An alternative to doing this two-step process of first writing to the local disk and then transfer the logs to Azure blob storage, is to write the log entries directly to Azure table storage (or in principal, to Azure blob storage for that matter). This is what I will do here.

Each log entry that the application writes will be a single row in a table in the Azure Table Storage. The log message itself and various meta data about it will be inserted into separate columns in the table. In order to achieve this, we first create a class that represents each entry in the table:

public class LogEntry : TableServiceEntity
{
    public LogEntry()
    {
        var now = DateTime.UtcNow;
        PartitionKey = string.Format("{0:yyyy-MM}", now);
        RowKey = string.Format("{0:dd HH:mm:ss.fff}-{1}", now, Guid.NewGuid());
    }
    #region Table columns
    public string Message { get; set; }
    public string Level { get; set; }
    public string LoggerName { get; set; }
    public string Domain { get; set; }
    public string ThreadName { get; set; }
    public string Identity { get; set; }
    public string RoleInstance { get; set; }
    public string DeploymentId { get; set; }
    #endregion
}

Note that the PartitionKey is the current year and month, and the RowKey is a combination of the date, time and a GUID. This is done to make the querying of the log entries efficient for our purpose. So, the next thing we need to do is to create a class that represents the table storage service. It needs to inherit from TableServiceContext:

internal class LogServiceContext : TableServiceContext
{
    public LogServiceContext(string baseAddress, StorageCredentials credentials) : base(baseAddress, credentials) {}
    internal void Log(LogEntry logEntry)
    {
        AddObject("LogEntries", logEntry);
        SaveChanges();
    }
    public IQueryable<LogEntry> LogEntries
    {
        get
        {
            return CreateQuery<LogEntry>("LogEntries");
        }
    }
}

The method that we will actually use in our code is the Log method which takes a LogEntry instance and persists it to table storage. What we need next, is to create a new Appender for Log4Net which interacts with the table store to store the log entries:

public class AzureTableStorageAppender : AppenderSkeleton
{
    public string TableStorageConnectionStringName { get; set; }
    private LogServiceContext _ctx;
    private string _tableEndpoint;
    public override void ActivateOptions()
    {
        base.ActivateOptions();
        var cloudStorageAccount =
            CloudStorageAccount.Parse(RoleEnvironment.GetConfigurationSettingValue(TableStorageConnectionStringName));
        _tableEndpoint = cloudStorageAccount.TableEndpoint.AbsoluteUri;
        CloudTableClient.CreateTablesFromModel(typeof(LogServiceContext), _tableEndpoint, cloudStorageAccount.Credentials);
        _ctx = new LogServiceContext(cloudStorageAccount.TableEndpoint.AbsoluteUri, cloudStorageAccount.Credentials);
    }
    protected override void Append(LoggingEvent loggingEvent)
    {
        Action doWriteToLog = () => {
            try
            {
                _ctx.Log(new LogEntry
                {
                    RoleInstance = RoleEnvironment.CurrentRoleInstance.Id,
                    DeploymentId = RoleEnvironment.DeploymentId,
                    Timestamp = loggingEvent.TimeStamp,
                    Message = loggingEvent.RenderedMessage,
                    Level = loggingEvent.Level.Name,
                    LoggerName = loggingEvent.LoggerName,
                    Domain = loggingEvent.Domain,
                    ThreadName = loggingEvent.ThreadName,
                    Identity = loggingEvent.Identity
                });
            }
            catch (DataServiceRequestException e)
            {
                ErrorHandler.Error(string.Format("{0}: Could not write log entry to {1}: {2}",
                    GetType().AssemblyQualifiedName, _tableEndpoint, e.Message));
            }
        };
        doWriteToLog.BeginInvoke(null, null);
    }
}

In the code above, the actually writing to the log is done asynchronically in order to prevent the log write to slow down the request handling. We are now done with all the coding. What is left is to use our new AzureTableStorageAppender. Here is the log4net.config:

<log4net>
  <appender name="AzureTableStoreAppender" type="Demo.Log4Net.Azure.AzureTableStorageAppender, Demo.Log4Net.Azure">
    <param name="TableStorageConnectionStringName" value="Log4Net.ConenctionString" />
  </appender>
  <root>
    <level value="DEBUG" />
    <appender-ref ref="AzureTableStoreAppender" />
  </root>
</log4net>

Notice the TableSTorageConnectionStringName attribute of the param element in the configuration. It corresponds to the property of the same name in the AzureTableStorageAppender. Furthermore, take take notice that it’s value is 'Log4Net.ConnectionString', which corresponds to a custom configuration setting that we will add to ServiceDefinition.csdef file:

<ServiceDefinition ...>
  <WebRole ...>
    <ConfigurationSettings>
      <Setting name="Log4Net.ConenctionString"/>
    </ConfigurationSettings>
    ...
  </WebRole>
</ServiceDefinition>

We also need to give the Log4Net.ConfigurationString setting a value in the ServiceConfiguration.cscfg file. It should be a connection string that points to the storage account to use for storing the log entries. In this example, let’s use the development storage:

<ServiceConfiguration ...>
  <Role ...>
    <ConfigurationSettings>
      <Setting name="Log4Net.ConenctionString" value="UseDevelopmentStorage=true"/>
    </ConfigurationSettings>
  </Role>
</ServiceConfiguration>

…and that’s it. You should now find the log entries in the table storage:

You can find the code for this example on GitHub. Suggestions for improvement are very welcome.