Samurai Programmer.com

Unless you’ve been hiding under the proverbial rock, you’ve probably seen the recent Pivot hoopla.  If you’re not familiar with it, it’s a way to visualize a large amount of data in a nice filterable format.  The nice thing about it is that it’s really easy to put together a pivot collection and there are a ton of tools available for just this purpose.  Just do a search on CodePlex for Pivot and you’ll get about 40’ish good results for tools you can use to create a Pivot Collection. 

So, I was putting together a proof-of-concept for an internal project and thought I would continue on with my series of blog posts on ASP.NET Error Message event logs with a post on how to visualize this data using a pivot.  You may wish to read parts 1 and 2 here:

• Part 1:  Parsing ASP.NET event log error messages for fun and profit
• Part 2:  Don’t guess when it comes to performance…

So, when I put together my pivot, I worked out a 3 step process:

1. Figure out what you want to Pivot
2. Find an API and convert the data
3. Generate and Test the collection

Let’s begin, shall we.

Figure out what you want to Pivot

The structure for the Pivot Collection is deceptively simple -

<?xml version="1.0"?> 
<Collection Name="Hello World Collection" …>
  <FacetCategories> 
    <FacetCategory Name="Hello World Facet Category One" Type="String"/> 
  </FacetCategories> 
  <Items ImgBase="helloworld.dzc"> 
    <Item Img="#0" Id="0" Href="http://www.getpivot.com" Name="Hello World!"> 
      <Description> This is the only item in the collection.</Description> 
      <Facets> 
        <Facet Name="Hello World Facet Category One"> 
         <String Value="Hello World Facet Value"/> 
        </Facet>       
      </Facets> 
    </Item> 
  </Items> 
</Collection>

The way that I think about the Items in the Collection are in the same way that you might think about an object.  For example, a Car object might have the following properties:

• Advertising blurb
• Car and Driver Reviews
• Color
• Make
• Model
• Engine
• 0-60mph time
• Max Speed

The common values like the Color, Make, Model, 0-60mph time and max speed become the facets or attributes that describe your object in relation to other instances of objects.  Things like the advertising blurbs and car and driver reviews or descriptions belong instead as properties of your Item directly in the Description element.

For our data, namely ASP.NET exceptions, we’re going to define an exception as the following:

• Item
• Name = Exception Type
• Description = Exception Message
• Facets
• Request Path
• Stack Trace
• Event Time
• Top Method of Stack Trace
• Top My Code Method of Stack Trace

This should allow us to group and drill through the common properties that might link exceptions together and still provide detailed error information when necessary.

Find an API and code it

The second step here is to find some code/API/tool that we can enhance for our purposes.  There are some great tools published by the Live Labs team – for example:

• Pivot Collection Tool for the Command Line
• Pivot Collection Tool for Microsoft Excel

While both tools could be used in this instance, in part 2 we found that some of our Event Logs we were parsing contained more than 10,000 items and I wanted a bit more control over how I converted the data.  “No touching” is the phrase of the day.  Fortunately, the command line tool was published on CodePlex with an API we can use.  Once you download the product you see that it contains 3 assemblies:

The last item there is the PauthorLib.dll which encapsulates many of the extension points within this great tool.  In-fact, it exposes about 7 different namespaces for our purposes:

For our purposes, we are going to focus on the Streaming set of namespaces.  Why?  Well, this is namely because we are going to be dealing with a lot of data and I didn’t want to load everything into memory before writing it to disk.  If you look at the contents of the Streaming namespace, you’ll see a great class called “AbstractCollectionSource”.  This looks fairly promising because it exposes two main methods:

class EventLogExceptionCollectionSource : AbstractCollectionSource
{
    protected override void LoadHeaderData()
    {
        throw new NotImplementedException();
    }

    protected override IEnumerable<PivotItem> LoadItems()
    {
        throw new NotImplementedException();
    }
}

Before we do anything, though, we need a constructor.  The constructor will be responsible for taking a string representing the path to our data and passing it to our base class’s constructor.

public EventLogExceptionCollectionSource(string filePath)
    : base(filePath)
{

    // Do nothing else.

}

Then, the first method, LoadHeaderData, is where we define our facets – Request Path, Stack Trace, etc.  - as well as the data types that each facet will be.  So, our code will be fairly simple and straight-forward:

protected override void LoadHeaderData()
{

    this.CachedCollectionData.FacetCategories.Add(
                                new PivotFacetCategory(STACKTRACE, 
                                                       PivotFacetType.LongString));
    this.CachedCollectionData.FacetCategories.Add(
                                new PivotFacetCategory(REQUESTPATH, 
                                                       PivotFacetType.String));
    this.CachedCollectionData.FacetCategories.Add(
                                new PivotFacetCategory(EVENTTIME, 
                                                       PivotFacetType.DateTime));
    this.CachedCollectionData.FacetCategories.Add(
                                new PivotFacetCategory(TOPMETHOD, 
                                                       PivotFacetType.String));
    this.CachedCollectionData.FacetCategories.Add(
                                new PivotFacetCategory(TOPAPPMETHOD, 
                                                       PivotFacetType.String));

    this.CachedCollectionData.Name = "Event Log Error Messages";


}

The second method, LoadItems(), is responsible for doing exactly what it suggests – this is where we load the data from whichever source we care about and then convert it into our PivotItem collection.  For our purposes, we’re going to load the XML file we defined in Part 1 of this series into a list of EventLogMessage objects and then convert those EventLogMessage objects into PivotItem objects:

protected override IEnumerable<PivotItem> LoadItems()
{
    // Load XML file
    XDocument document = XDocument.Load(this.BasePath);

    // Populate collection of EventLogMessage objects
    var messages = from message in document.Descendants("Message")
                   select EventLogMessage.Load(message.Value);

    int index = 0;
    foreach (EventLogMessage message in messages)
    {

        PivotItem item = new PivotItem(index.ToString(), this);
        item.Name = message.Exceptiontype;
        item.Description = message.Exceptionmessage;
        item.AddFacetValues(REQUESTPATH, message.Requestpath);
        item.AddFacetValues(STACKTRACE, message.Stacktrace);
        item.AddFacetValues(EVENTTIME, message.Eventtime);
        item.AddFacetValues(TOPMETHOD, message.StackTraceFrames[0].Method);
        item.AddFacetValues(TOPAPPMETHOD, GetFirstNonMicrosoftMethod(message.StackTraceFrames));

        index++;
        yield return item;

    }
}

The key method calls from above are the AddFacetValues(…) method calls.  This method essentially sets the attributes we wish to have our data pivot upon.  This, by itself, isn’t enough to generate our great pivot calls – we need to call our code from somewhere.  Since this is a simple app, we’re going to make it a console app.  For our Collection to get generated we need to use a few other objects included in this API:

• EventLogExceptionCollectionSource – The class we created above.
• HtmlImageCreationSourceFilter – This class will generate the tile in the Pivot based upon some HTML template we specify.
• LocalCxmlCollectionTarget – Generates the Collection XML file at the path we specify.
• DeepZoomTargetFilter – Generates the deep zoom files to support our collection XML file and also enables all of our fancy transitions.

In practice, the code is pretty simple and straight forward and I applaud the people who wrote this library:

private static void GenerateExceptionPivot(string inputFile, string outputFolder)
{
    string collectionName = Path.Combine(outputFolder, "MyExceptions.cxml");

    EventLogExceptionCollectionSource source = 
                    new EventLogExceptionCollectionSource(inputFile);
    HtmlImageCreationSourceFilter sourceFilter1 = 
                    new HtmlImageCreationSourceFilter(source);
    sourceFilter1.HtmlTemplate = 
                    "<html><body><h1>{name}</h1>{description}</body></html>";
    sourceFilter1.Width = 600;
    sourceFilter1.Height = 600;

    LocalCxmlCollectionTarget target = 
                    new LocalCxmlCollectionTarget(collectionName);
    DeepZoomTargetFilter targetFilter1 = 
                    new DeepZoomTargetFilter(target);
    targetFilter1.Write(sourceFilter1);

}

That last statement, targetFilter1.Write(…) is what will actually execute everything and write our resultant files to disk.

Generate and Test the collection

So, now if we run our console application and call that GenerateExceptionPivot(…) method, we’ll get some great output.

What’s nice about the Library is that it provides progress as it iterates through your data (in the red rectangle) and also in the blue rectangle, we need that it’s multi-threaded by default.  This is primarily for the most intensive part of the operation – the creation of the deep zoom artifacts.  If you have one of those new fangled machines with 2+ cores, you can tweak the number of threads that it will spawn for this operation by setting the ThreadCount property of the DeepZoomTargetFilter object.  This may or may not improve your performance but it’s nice that the option is available.

...
DeepZoomTargetFilter targetFilter1 = 
                new DeepZoomTargetFilter(target);

targetFilter1.ThreadCount = 100;
targetFilter1.Write(sourceFilter1);
...

Once our collection has been generated, we can browse it in an explorer.exe window just to get an idea of what our code has wrought:

And then to test it, you can just point the Live Labs Pivot application at our “MyExceptions.cxml” file and view the wonderful data.  For example, you can look at the Event Time in the histogram view to see how your exceptions broke down over time.  You can also filter your data by things like the RequestPath (the page that threw the exception) or the Method that was at the top of the callstack.

Then, you can zoom in on a specific time slice you care about:

Then, if you want to view the details for a specific instance, just click the corresponding tile.  Then, a new side bar will appear on the right hand side with all of the details we stored in this record:

We generated a single collection in this blog post.  One thing to keep in mind is that each collection should have no more than 3,000 items.  For collections in which you want to have more than 3,000 items, you should look at potentially creating a Linked Collection.  That will be the subject of an upcoming blog post. 

Until next time!