Increase IIS Logs to SQL Log Parser Import Performance w/ transactionRowCount

A few years ago I showed how to use Microsoft’s Log Parser tool to take IIS log files and import into a SQL database.

From Microsoft: Log parser is a powerful, versatile tool that provides universal query access to text-based data such as log files, XML files and CSV files, as well as key data sources on the Windows® operating system such as the Event Log, the Registry, the file system, and Active Directory®. You tell Log Parser what information you need and how you want it processed. The results of your query can be custom-formatted in text based output, or they can be persisted to more specialty targets like SQL, SYSLOG, or a chart.

In short, using something like this to take IIS logs and dump into a new SQL table:

C:\Program Files (x86)\Log Parser 2.2>logparser “SELECT * INTO iisLogs FROM c:\temp\logs\*.log ” -i:iisw3c -o:SQL -server:localhost -database:webLogs -username:sa -password:yourpass -createTable: ON

But, if you’re importing tons of records, it might seem to take a while. BUT: you can use the option “transactionRowCount” to gain some performance. The transactionRowCount option determines how many rows are included in each transaction. By default, transactionRowCount is 1, so after every row, the transaction is committed. If you set it to “-1” it will include everything in 1 large transaction.

Test Results

Below a did a few tests. My test included 36,000,000 rows.

Using the default (no need to supply the option):

Statistics:
-----------
Elements processed: 36158727
Elements output: 36158727
Execution time: 8448.14 seconds (02:20:48.14)

What if we only use 1 entire transaction (“transactionRowCount:-1”)

Statistics:
-----------
Elements processed: 36158727
Elements output: 36158727
Execution time: 3944.09 seconds (01:05:44.09)

If we use a transaction every 10,000 records (“transactionRowCount:10000”)

Statistics:
-----------
Elements processed: 36158727
Elements output: 36158727
Execution time: 3818.74 seconds (01:03:38.74)

Results

In my case, the 10,000 value was just slightly faster than -1.

The usage of this option:

logparser "SELECT * INTO iisLogs4 FROM c:\logs\*.log" -i:iisw3c -o:SQL -server:localhost -database:iisLogs -transactionRowCount:10000 -createTable:ON

This wasn’t an extensive test by any means, but I figured it’s a good starting point for anyone looking to speed up their imports.

 

Increase IIS Logs to SQL Log Parser Import Performance w/ transactionRowCount

One thought on “Increase IIS Logs to SQL Log Parser Import Performance w/ transactionRowCount

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s