An inspection of the execution plan showed that one part of the procedure was doing an index scan and that it was this that was taking up over 85% of the execution time. I was totally baffled, it was hitting the index so why wasn’t it faster?

Then I learned three things all at once:

• Implicit Conversion
• non-sargable kills performance
• Index Seek is preferred of Index Scan

Implicit Conversion

In the stored proc a variable was defined as an NVARCHAR(20) but the field in the table it was being compared to was a CHAR(10). This lead to an implicit conversion of the variable to a CHAR(10), which lead into the next issue:

Non-Sargable Kills Performance

Sargable refers to the pseudo-acronym “SARG” – Search ARGument and refers to a WHERE clause that compares a constant value to a column value. The implicit conversion was causing a non-SARGable condition which means the WHERE clause cannot use an index.

Index Scan

Because of the non-sargable condition an Index Scan was being performed. An Index Scan is just as bad as a Table Scan in the SQL realm and should be avoided at all costs.

The solution was simple: Change the variable to a CHAR(10). After doing that the Index Scan became an Index Seek and the whole stored procedure returned in less than a second. Any time I see an order of magnitude improvement like that from one simple change it just boggles my mind.

The table listed below is for use with the Activity table and not the Balance table. Either a separate matrix table will have to be created or an additional field would have to be added for calculating account formulas with the Balance table.

Pd_Type

The Period Type is used to indicate the type of formula being used. The types listed are:

• Month: The given months activity

• Quarter: The given quarters activity

• Semiannual: The given half-year activity

• Annual: The entire years activity

• Other types that could be created include a months or quarter ending balance.

The Period ID and Name are used to identify a given time frame:

: The month number (1 thru 12) and name (January thru December)
• Quarter: The quarter number (1 thru 4) and name (QTR1 thru QTR4)
• Semiannual: The semiannual number (1 or2) and name (SA1 and SA2)
• Annual: The year number (1) or name (YR)

pd_1 thru pd_12

The values for period 1 through 12 can only be 1 or 0. These fields are multiplied with the Activity table month fields to create the necessary formulas.

Creating Formulas

To create a formula using the Period Matrix and the Activity table you would create a select statement where in the period fields in the Activity table were multiplied to the period fields in the Matrix table then add them all together:

( Activity.Jan * Pd_Matrix.pd_1 + Activity.Feb * Pd_Matrix.pd_2
Activity.Mar * Pd_Matrix.pd_3 + Activity.Apr * Pd_Matrix.pd_4
Activity.May * Pd_Matrix.pd_5 + Activity.Jun * Pd_Matrix.pd_6
Activity.Jul * Pd_Matrix.pd_7 + Activity.Aug * Pd_Matrix.pd_8
Activity.Sep * Pd_Matrix.pd_9 + Activity.Oct * Pd_Matrix.pd_10
Activity.Nov * Pd_Matrix.pd_11 + Activity.Dec * Pd_Matrix.pd_12)

In the WHERE clause of the select statement you would set the Period Type (Pd_Type) and Period Name (Pd_Name) of the Matrix table to the name of the formula. For example, if you wanted to show the monthly activity for January you would create the following select statement:

SELECT
      Activity.AcctID
     , Activity.DeptID
     , Activity.Class
     , Pd_Matrix.Pd_Name
     , Activity.Year
     , ( Activity.Jan * Pd_Matrix.pd_1 + Activity.Feb * Pd_Matrix.pd_2
         + Activity.Mar * Pd_Matrix.pd_3 + Activity.Apr * Pd_Matrix.pd_4
         + Activity.May * Pd_Matrix.pd_5 + Activity.Jun * Pd_Matrix.pd_6
         + Activity.Jul * Pd_Matrix.pd_7 + Activity.Aug * Pd_Matrix.pd_8
         + Activity.Sep * Pd_Matrix.pd_9 + Activity.Oct * Pd_Matrix.pd_10
         + Activity.Nov * Pd_Matrix.pd_11 + Activity.Dec * Pd_Matrix.pd_12
         ) As Amount

FROM Activity, Pd_Matrix

WHERE
         (Pd_Type = ‘month’)
     And (Pd_Matrix.Pd_Name = ‘JAN’)

This SQL statement could be changed to a parameterized stored procedure with variables for Pd_Type and Pd_Name. This would allow you to return a recordset for any of the given formulas.

Table Folding

The Period Matrix can also be used in table folding, where the columns of the original table are converted to rows. This is accomplished by creating a Cartesian product between the original table, in our case the Activity table, and the Period Matrix table.

A Cartesian product result set, also known as a Cross Join, is familiar to people that have created SQL statements with bad or missing joins between tables. They are recognizable because the result set will have the same row multiple times or the total row count far exceeds the total number of expected rows. In most cases this is unwanted but it comes in very handy for table folding.

The following SQL select statement will fold the Activity table’s period activity columns into rows:

SELECT

       Activity.AcctID
     , Activity.DeptID
     , Activity.Class
     , Pd_Matrix.Pd_Name
     , Activity.Year
     ,    (Activity.Jan * Pd_Matrix.pd_1 + Activity.Feb * Pd_Matrix.pd_2
         + Activity.Mar * Pd_Matrix.pd_3 + Activity.Apr * Pd_Matrix.pd_4
         + Activity.May * Pd_Matrix.pd_5 + Activity.Jun * Pd_Matrix.pd_6
         + Activity.Jul * Pd_Matrix.pd_7 + Activity.Aug * Pd_Matrix.pd_8
         + Activity.Sep * Pd_Matrix.pd_9 + Activity.Oct * Pd_Matrix.pd_10
         + Activity.Nov * Pd_Matrix.pd_11 + Activity.Dec * Pd_Matrix.pd_12
         ) As Amount

FROM Activity, Pd_Matrix
 
WHERE Pd_Type = ‘month’

Note that this is identical to the SQL used to show the activity for just one month used previously, with the only exception being in the WHERE clause. Instead of indicating which month (Pd_Name) to show, the SQL was allowed to return all the available months.

This statement works for the same reasons the previous SQL statement worked. When the Pd_Name is JAN then pd_1 equals one while the rest of the pd_2 through pd_12 equals zero. When the multiplications are carried out all the results are all zero except for January’s, which is equal to the January activity. When the addition is carried out therefore only the January amount is returned.

Also, since there is no join between the two tables the result set will have twelve times the number of rows that the Activity table actually has. For each row in the Activity table there will be twelve rows generated in the result because there are twelve possible rows for the Pd_Type of ‘month’. If instead of choosing ‘month’ we had chosen ‘quarter’ for the Period Type then the results set would have four times the number of rows in the Activity table.

This method of table folding is fast and very simple to create and maintain. The most common way table folding is done though is by manually creating the final table then creating an append or update query for each column to fold. In the case of our example we would have had to create and execute thirteen queries in order to obtain the same results.

Table Folding and Pivoting Using Formulas

Where as table folding is deals with turning columns into rows, table pivoting is turning rows into columns.

Taking the example from the table folding section lets add the requirement that our resultset show the Actual and Budget amounts in separate columns. The periods will still be shown as rows.

The SQL needed in Access would be:

SELECT

       Activity.AcctID
     , Activity.DeptID
     , Pd_Matrix.Pd_Name
     , Activity.Year
     ,Sum( IIF(Activity.Class = “Actual”,
         (Activity.Jan * Pd_Matrix.pd_1 + Activity.Feb * Pd_Matrix.pd_2
         + Activity.Mar * Pd_Matrix.pd_3 + Activity.Apr * Pd_Matrix.pd_4
         + Activity.May * Pd_Matrix.pd_5 + Activity.Jun * Pd_Matrix.pd_6
         + Activity.Jul * Pd_Matrix.pd_7 + Activity.Aug * Pd_Matrix.pd_8
         + Activity.Sep * Pd_Matrix.pd_9 + Activity.Oct * Pd_Matrix.pd_10
         + Activity.Nov * Pd_Matrix.pd_11 + Activity.Dec * Pd_Matrix.pd_12
         ), 0)) As Actual

     ,Sum( IIF(Activity.Class = “Budget”,
         (Activity.Jan * Pd_Matrix.pd_1 + Activity.Feb * Pd_Matrix.pd_2
         + Activity.Mar * Pd_Matrix.pd_3 + Activity.Apr * Pd_Matrix.pd_4
         + Activity.May * Pd_Matrix.pd_5 + Activity.Jun * Pd_Matrix.pd_6
         + Activity.Jul * Pd_Matrix.pd_7 + Activity.Aug * Pd_Matrix.pd_8
         + Activity.Sep * Pd_Matrix.pd_9 + Activity.Oct * Pd_Matrix.pd_10
         + Activity.Nov * Pd_Matrix.pd_11 + Activity.Dec * Pd_Matrix.pd_12
         ), 0)) As Budget

FROM Activity, Pd_Matrix

WHERE Pd_Type = ‘month’

GROUP BY
       Activity.AcctID
     , Activity.DeptID
     , Pd_Matrix.Pd_Name
     , Activity.Year

In SQL Server the CASE statement would replace the IIF( ) function:

SELECT

       Activity.AcctID
     , Activity.DeptID
     , Pd_Matrix.Pd_Name
     , Activity.Year
    
     ,Sum( CASE Activity.Class
         WHEN “Actual”THEN
         (Activity.Jan * Pd_Matrix.pd_1 + Activity.Feb * Pd_Matrix.pd_2
         + Activity.Mar * Pd_Matrix.pd_3 + Activity.Apr * Pd_Matrix.pd_4
         + Activity.May * Pd_Matrix.pd_5 + Activity.Jun * Pd_Matrix.pd_6
         + Activity.Jul * Pd_Matrix.pd_7 + Activity.Aug * Pd_Matrix.pd_8
         + Activity.Sep * Pd_Matrix.pd_9 + Activity.Oct * Pd_Matrix.pd_10
         + Activity.Nov * Pd_Matrix.pd_11 + Activity.Dec * Pd_Matrix.pd_12
         ) ELSE 0 END) Actual

     ,Sum( CASE Activity.Class
         WHEN “Budget”THEN
         (Activity.Jan * Pd_Matrix.pd_1 + Activity.Feb * Pd_Matrix.pd_2
         + Activity.Mar * Pd_Matrix.pd_3 + Activity.Apr * Pd_Matrix.pd_4
         + Activity.May * Pd_Matrix.pd_5 + Activity.Jun * Pd_Matrix.pd_6
         + Activity.Jul * Pd_Matrix.pd_7 + Activity.Aug * Pd_Matrix.pd_8
         + Activity.Sep * Pd_Matrix.pd_9 + Activity.Oct * Pd_Matrix.pd_10
         + Activity.Nov * Pd_Matrix.pd_11 + Activity.Dec * Pd_Matrix.pd_12
         ) ELSE 0 END) Budget

FROM Activity, Pd_Matrix

WHERE Pd_Type = ‘month’
 
GROUP BY
       Activity.AcctID
     , Activity.DeptID
 
     , Pd_Matrix.Pd_Name
     , Activity.Year
 

Summary

The Period Matrix table is a versatile tool for SQL development. It can be used in all DBMSs with little modification to the SQL statements. It can be used to replace several separate SQL statements needed to create the different formulas and it is useful in table folding and pivoting.

Activity Table

The table Activity used in this example has the following layout:

The AcctID and DeptID refer to account and department numbers, which are a part of our fictitious company’s accounting key. The Class refers to the type of monies, in our examples we will stick to only Actual and Budget dollar amounts. The Year is the calendar year in which the values apply. The BeginBalance is the current Year’s beginning balance while the fields Jan through Dec each contain the activity for the given month.

The data in the table was generated using the following criteria:

1. The current calendar year is 1999.
2. The current accounting period is June.
3. There are two years of history, so years 1997 and 1998 have actual amounts only.
4. There is a budget for years 1999 and 2000.
5. There are no actual amounts for the year 2000, only budget amounts.

Budget Variance and History

In our first example we will show two techniques of producing a very common accounting scenario: generating a report for a given accounting period showing the comparison between the actual and budgeted amounts, along with history. As an added twist we will show the next years budget also. For this example we will further limit it to show January 1999 as the current accounting period.

The result we produce will need to show the account-department number, January 1997, 1998 and 1999 actual amounts, January 1999 and 2000 budget amounts.

As can be seen, we are pivoting the Actual table on two fields, Year and Class. We will employee several techniques to perform this pivoting to show that there is more than one way to do it. The techniques we will use are:

• Select query with the IIF( ) function
• Select query with a combination of the IIF( ) and CHOOSE( ) functions

Pivot using IIF( ) Function

The IIF() function is a built in MS Access function. It is identical to the IF() function in Excel and has the following syntax:

IIF( [logical test], [value if true], [value if false] )

The logical test can be any type of expression that returns a true or false value.

As a simple example, say we wanted to select the Class field from the Activity table only the word “actual”. If the Class is “actual” we want to see “Real Dollars”, otherwise show “Not Real Dollars”. To do this we would create the following query:

SELECT Activity.AcctID, Activity.DeptID, IIf([Class]="actual","Real Dollars","Not Real Dollars") AS RealDollars, Activity.BeginBalanceFROM Activity;

This would return the beginning balances with the either “Real Dollars” or “Not Real Dollars” in the RealDollars field. An example output is:

Now let us turn our attention back to the pivot table problem. The problem is to show for the years 1997 through 1999 the actual dollar amounts (Class = actual), and for the years 1999 and 2000 the budgeted amounts (Class = budget).

Looking at the data, we know that for the years 1997 and 1998 there are no budget amounts. Because of this we can set up two fields in our query using the IIF( ) function and the table field [Year]:

[1997] = IIF( [Year] = "1997", [Jan], 0)[1998] = IIF( [Year] = "1998", [Jan], 0)

With these two functions the given query field value will either be the given years January actual amount or zero.

The next two years, 1999 and 2000, have both actual and budget amounts. However, since we are currently in the calendar year 1999 the year 2000 will have only budget amounts. Using this we can create fields for 1999 and 2000:

[Actual 1999] = IIF ([Year]="1999" AND [Class] = "actual", [Jan], 0)[Budget 1999] = IIF ([Year]="1999" AND [Class] = "budget", [Jan], 0)[Budget 2000] = IIF ([Year]="2000" AND [Class] = "budget", [Jan], 0)

These functions will return their indicated values, January Actual 1999, Budget 1999 or Budget 2000, or they will return a zero.

Combining these into a final select query we get:

SELECT act.AcctID, act.DeptID, IIf([act].[Year]="1997",[act].[Jan],0) AS 1997, IIf([act].[Year]="1998",[act].[Jan],0) AS 1998, IIf([act].[Year]="1999" And [act].[Class]="actual",[act].[Jan],0) AS Actual1999, IIf([act].[Year]="1999" And [act].[Class]="budget",[act].[Jan],0) AS Budget1999, IIf([act].[Year]="2000" And [act].[Class]="actual",[act].[Jan],0) AS Actual2000, IIf([act].[Year]="2000" And [act].[Class]="budget",[act].[Jan],0) AS Budget2000FROM Activity AS act;

As a result for account 1000 we get:

As you can see there are a lot of zeros. It almost looks like we accomplished the goal of the problem. We have columns with only 1997 values, or 1998 values, or Actual 1999 values, etc, which means we have successfully pivoted on the desired fields, but this result set is quit useless as it is. What we need now is a way to reduce the rows down so that true comparisons can be made between the history and the budget.

This is accomplished by using the aggregate function sum() in the query. Rewriting the query and summing on the amount columns we generate the following select query:

SELECT act.AcctID, act.DeptID, Sum(IIf([act].[Year]="1997",[act].[Jan],0)) AS 1997, Sum(IIf([act].[Year]="1998",[act].[Jan],0)) AS 1998, Sum(IIf([act].[Year]="1999" And[act].[Class]="actual",[act].[Jan],0)) AS Actual1999, Sum(IIf([act].[Year]="1999" And[act].[Class]="budget",[act].[Jan],0)) AS Budget1999, Sum(IIf([act].[Year]="2000" And[act].[Class]="budget",[act].[Jan],0)) AS Budget2000FROM Activity AS actGROUP BY act.AcctID, act.DeptID;

This query returns the results we can use:

By grouping on the AcctID and DeptID and summing on the other columns we have effectively pivoted the table on two columns, [Year] and [Class].

Looking at the first result set (without the sum()) we see that for AcctID 1000, DeptID 1000 there is one row for each non-zero column value (this is assuming that no original value was zero to begin with). What is not seen is the fact that this result is really a snap shot of the original table but without showing the [Year] and [Class] columns, these are hidden in the IIF( ) function.

By comparing the original table (below) to the two above we can see how the values where first pivoted on the [Year] field (first query) then summed to pivot on the [Class] field.

Also, note that this was a single select query as opposed to the use of two crosstabs combined in a select query.

Pivot Using IIF( ) and CHOOSE( ) Functions

From the previous examples we see how to pivot a table on two columns. In all cases though, we were only able to show data for one accounting period, January. In order to show any of the other months activity using the above techniques we would have to create eleven more queries. Maintaining twelve queries, which only shows the data in one way is very cumbersome and over time as the number of reports needed increased, impossible to maintain.

The solution would be to write a single query with a parameter to be used in all twelve cases. The parameter would be the number of the month whose values we want to see. If we were to stay with just using the IIF( ) function our query would have fields like this:

[1997] = IIF([Year]="1999", (IIF([Month]=1, [JAN], (IIF([Month]=2, [FEB], …0) 0) 0)

This again would be impossible to maintain.

The alternative is to use the CHOOSE() function. The syntax is:

 Choose (index, choice-1[, choice-2, ... [, choice-n]])

Where index refers to the choice number. If index equals 1 then choice-1 is returned, if it is 2 then choice-2 is returned, on down to index equal n then choice-n is returned.

Using our parameter we can rewrite the above function as:

 [1997] = IIF([Year]="1997", Choose([Month], [JAN], [FEB], …,[DEC]), 0)

PARAMETERS MonthNum Short;SELECT act.AcctID, act.DeptID, Sum(IIf([act].[Year]="1997",	Choose([MonthNum]		,[act].[Jan],[act].[Feb],[act].[Mar]		,[act].[Apr],[act].[May],[act].[Jun]		,[act].[Jul],[act].[Aug],[act].[Sep]		,[act].[Oct],[act].[Nov],[act].[Dec]),0)) AS 1997, Sum(IIf([act].[Year]="1998",	Choose([MonthNum]		,[act].[Jan],[act].[Feb],[act].[Mar]		,[act].[Apr],[act].[May],[act].[Jun]		,[act].[Jul],[act].[Aug],[act].[Sep]		,[act].[Oct],[act].[Nov],[act].[Dec]),0)) AS 1998, Sum(IIf([act].[Year]="1999" And [act].[Class]="actual",	Choose([MonthNum]		,[act].[Jan],[act].[Feb],[act].[Mar]		,[act].[Apr],[act].[May],[act].[Jun]		,[act].[Jul],[act].[Aug],[act].[Sep]		,[act].[Oct],[act].[Nov],[act].[Dec]),0)) AS Actual1999, Sum(IIf([act].[Year]="1999" And [act].[Class]="budget",	Choose([MonthNum]		,[act].[Jan],[act].[Feb],[act].[Mar]		,[act].[Apr],[act].[May],[act].[Jun]		,[act].[Jul],[act].[Aug],[act].[Sep]		,[act].[Oct],[act].[Nov],[act].[Dec]),0)) AS Budget1999, Sum(IIf([act].[Year]="2000" And [act].[Class]="budget",	Choose([MonthNum]		,[act].[Jan],[act].[Feb],[act].[Mar]		,[act].[Apr],[act].[May],[act].[Jun]		,[act].[Jul],[act].[Aug],[act].[Sep]		,[act].[Oct],[act].[Nov],[act].[Dec]),0)) AS Budget2000FROM Activity AS actGROUP BY act.AcctID, act.DeptID;

CrossTab Queries

CrossTab queries are an easy way in MS Access to pivot tables. They are limited in that you can only pivot on one field and can only aggregate one field.

As an example we look at Activity table in the Sample.mdb. If we were asked to show the January activity for each account – department combination for each year using a crosstab query, we would create the following SQL:

   TRANSFORM Sum(Activity.Jan) AS SumOfJan
   SELECT Activity.AcctID, Activity.DeptID
   FROM Activity
   WHERE (Activity.Class = “actual”)
   GROUP BY Activity.AcctID, Activity.DeptID
   PIVOT Activity.Year In (1997,1998,1999,2000);

The TRANSFORM identifies the column to aggregate and the type of aggregation (sum). Other types of aggregation possible are avg, min, max, count, etc.

The PIVOT identifies which column is pivoted on. In this case it is the Year column, the IN clause indicates that we only want to see these four specific column headings. If the IN clause was not specified then all possible values of Year would be used as column headings.

In the Query Builder, the Row Headings indicate which fields to show in the results, while the Column Heading shows the pivot field and the Value is the aggregated field. For a crosstab there is no limit to the number of Row Headings but you need one Column Heading and one Value. Of course with a crosstab you can have only one Column Heading and one Value.

The first eight rows of the result set would be:

As can be seen this is a very simple query, easy to setup and understand. Simply identify the column to pivot on and what column to aggregate and that is it. The problems start to arise whenever more than one pivot value or transform column is needed.

Pivoting on Two Columns – Multiple Value Field Query

Single CrossTab queries cannot pivot on more than one column. However, situations arise frequently where it is needed to pivot on two or more columns. One workaround for this, as documented in the Microsoft Knowledge Base (Q109939), is to build separate crosstab queries using the same underlying table, each pivoting on the same column but showing only specific values from the second column. The final result is obtained by joining the crosstab queries in a Select query. Microsoft refers to this as a Multiple Value Field query.

This method of joining crosstab queries is simple to implement whenever the number of columns to pivot on is small. However, as the number of column pivots increases the number of required joins also increases. Eventually a point is reached where the query is too complex to maintain or even to complex to execute. A rule of thumb should be that if it is needed to pivot on more than two columns then crosstab queries should not be used.

To create the Budget Variance and History report with a crosstab we will need to use a Multiple Value Field query. To accomplish this we use the previous crosstab query for our actual amounts and create the following crosstab for our budget amounts:

   TRANSFORM Sum(Activity.Jan) AS SumOfJan
   SELECT Activity.AcctID, Activity.DeptID
   FROM Activity
   WHERE (Activity.Class = “budget”)
   GROUP BY Activity.AcctID, Activity.DeptID
   PIVOT Activity.Year In (1997,1998,1999,2000);

As you can see the only difference is the WHERE clause.

Next we create the select query using both of the crosstab queries as the source. We join on the account and department numbers:

   SELECT
      actual.AcctID, actual.DeptID,

      actual.[1997], actual.[1998],
      actual.[1999], budget.[1999]
      actual.[2000], budget.[2000]
   FROM
      [CrossTab Actual by Year] AS actual
      INNER JOIN [CrossTab Budget by Year] AS budget
         ON (actual.DeptID = budget.DeptID)
         AND (actual.AcctID = budget.AcctID);

In this case since we are currently in the year 1999 we would not have budget numbers for 1997 or 1998. The query builder looks like:

The results from this are:

From this result set it is easy to directly compare the actual amounts to the budgeted amounts. Of course it must be remembered that these amounts are from just one month and as can be guessed trying to compare two separate months from the same year using a crosstab in this manner is cumbersome.

Using crosstabs for simple queries is easy. However for typical reporting needs crosstabs become too unwieldy and difficult to maintain. Fortunately we do have some alternatives.

]]> http://blog.ddpruitt.net/2004/09/25/ms-access-pivots-part-i/feed/ 0