Increase IIS Logs to SQL Log Parser Import Performance w/ transactionRowCount

March 1, 2018March 1, 2018 chrisbittingiis, logs3 Comments

A few years ago I showed how to use Microsoft’s Log Parser tool to take IIS log files and import into a SQL database.

From Microsoft: Log parser is a powerful, versatile tool that provides universal query access to text-based data such as log files, XML files and CSV files, as well as key data sources on the Windows® operating system such as the Event Log, the Registry, the file system, and Active Directory®. You tell Log Parser what information you need and how you want it processed. The results of your query can be custom-formatted in text based output, or they can be persisted to more specialty targets like SQL, SYSLOG, or a chart.

In short, using something like this to take IIS logs and dump into a new SQL table:

C:\Program Files (x86)\Log Parser 2.2>logparser “SELECT * INTO iisLogs FROM c:\temp\logs\*.log ” -i:iisw3c -o:SQL -server:localhost -database:webLogs -username:sa -password:yourpass -createTable: ON

But, if you’re importing tons of records, it might seem to take a while. BUT: you can use the option “transactionRowCount” to gain some performance. The transactionRowCount option determines how many rows are included in each transaction. By default, transactionRowCount is 1, so after every row, the transaction is committed. If you set it to “-1” it will include everything in 1 large transaction.

Test Results

Below a did a few tests. My test included 36,000,000 rows. Continue reading “Increase IIS Logs to SQL Log Parser Import Performance w/ transactionRowCount” →

How To Create A Simple .Net Core (c#) AWS Lambda Function – Start to Finish

December 29, 2017 chrisbittingaws, lambdaLeave a comment

Creating a simple, serverless app w/ AWS Lambda is fairly easy, but some documentation out there is outdated or using the preview toolkit. Below are some steps that show how to do this today, in a few steps. I’m using Visual Studio 2017. I’m also going to assume you already have your AWS credentials on your machine, if not, that’s a different topic.

Install the AWS Toolkit for Visual Studio 2017 for Visual Studio. This is required to give you the project templates.
Open Visual Studio and start a new project, choose “AWS Lambda Project” and give your project a name (I picked “awsLambdaTest”)
At the Blueprint choice, choose “empty” then click “finish”.
Your project will now create after a few seconds and should look like this:
You can edit your code (in Function.cs)
If you don’t edit anything and publish
Now give your function a name and choose “next”:
Now choose a role, the lambda_exec role is fine, then click “upload”.
By default, the Lambda Function view will appear – this allows you to test your function. If you enter a string in the box under “sample input” and press “invoke”, you’ll see your function response.

Creating Valid ZIP Archives of Multiple Files in C# / .Net

October 26, 2017October 26, 2017 chrisbitting.net, aws, csharp, elastic beanstalk, folder, zip5 Comments

Are .zip files ever going away? I remember back in the 90’s using WinZip as an alternative to the PKZip command line option. Anyway, fast forward 20+ years, and ZIP is still common and a great way to package files. Long story short: AWS Elastic Beanstalk allows you to easily deploy apps using a .zip file (if you haven’t tried Elastic Beanstalk – it’s pretty awesome) and I wanted a faster way to create a .zip of an app. (Yes, I know it’s not the best way to deploy like Git or the API).

There is a bunch of great sample code out there for creating ZIP archives in c# .Net using the ZipArchive Class in System.IO.Compression, but nothing seems to be a complete sample, showing multiple files. Below is what I’ve been using. One difference in this is changing the path separators from backslashes to forward-slashes. Without this, AWS wasn’t able to extract my .zip archive. I would see errors such as: Continue reading “Creating Valid ZIP Archives of Multiple Files in C# / .Net” →

Using Amazon Polly from .net / c#, Get MP3 File

April 7, 2017April 7, 2017 chrisbitting.net, aws, mp3, tts5 Comments

knight-rider-car-kitt If you haven’t checked out Amazon’s new Polly (Text-to-Speech (TTS) cloud service) it does produce some pretty great, life-like audio. Below is a quick sample on how you can feed some text to Polly, and get an MP3 file with it. I don’t cover all of the install options of the AWS Toolkit for Visual Studio / .Net, but it’s pretty simple. (listen to this text here)

Below is some code:

AmazonPollyClient pc = new AmazonPollyClient();

SynthesizeSpeechRequest sreq = new SynthesizeSpeechRequest();
sreq.Text = "Your Sample Text Here";
sreq.OutputFormat = OutputFormat.Mp3;
sreq.VoiceId = VoiceId.Amy;
SynthesizeSpeechResponse sres = pc.SynthesizeSpeech(sreq);

using (var fileStream = File.Create(@"c:\yourfile.mp3"))
{
sres.AudioStream.CopyTo(fileStream);
fileStream.Flush();
fileStream.Close();
}

Also make sure you have the below included:

using Amazon.Polly;
using Amazon.Polly.Model;

And these 2 NuGet packages added to your project:

aws-polly

This only scratches the surface of what Polly can do. Between streaming, SSML, Lexicons and more at a great price, I think we’ll be seeing more applications use this.

Sorting ExpandoObject / Dynamic Object Lists in c#

February 10, 2017 chrisbittingdynamic, expando, ilist, linq, listLeave a comment

Sometimes I find myself using lists of ExpandoObjects to quickly create lists of dynamic objects. They are super fast to create, and so flexible. You may however want to sort the list using one of the dynamic fields you added. Below is an easy way I use:

yourlist.OrderBy(x => ((IDictionary<string, object>)x)[“yourfield”])

Below is a complete example, making use of a couple Windows versions for some fun sample data.

using System;
using System.Collections.Generic;
using System.Dynamic;
using System.Linq;

namespace ExpandoObjectSort
{
    internal class Program
    {
        private static void Main(string[] args)
        {
            IList<ExpandoObject> windows = new List<ExpandoObject>();

            //add some data
            dynamic win31 = new ExpandoObject();
            win31.codename = "Janus";
            win31.windows = "Windows 3.1";
            win31.released = new DateTime(1992, 4, 6);
            windows.Insert(windows.Count, win31);

            dynamic win95 = new ExpandoObject();
            win95.codename = "Chicago";
            win95.windows = "Windows 95";
            win95.released = new DateTime(1995, 8, 24);
            windows.Insert(windows.Count, win95);

            dynamic winXP = new ExpandoObject();
            winXP.codename = "Whisler";
            winXP.windows = "Windows XP";
            winXP.released = new DateTime(2001, 10, 25);
            windows.Insert(windows.Count, winXP);

            dynamic win8 = new ExpandoObject();
            win8.codename = "Blue";
            win8.windows = "Windows 8";
            win8.released = new DateTime(2012, 10, 26);
            windows.Insert(windows.Count, win8);

            //loop through the list:
            Console.WriteLine("[default]");
            foreach (dynamic win in windows)
            {
                Console.WriteLine(win.windows + " - Codename:" + win.codename + " - Released: " + ((DateTime)win.released).ToShortDateString());
            }

            //sort via codename
            Console.WriteLine("[codename]");
            foreach (dynamic win in windows.OrderBy(x => ((IDictionary<string, object>)x)["codename"]))
            {
                Console.WriteLine(win.windows + " - Codename:" + win.codename + " - Released: " + ((DateTime)win.released).ToShortDateString());
            }

            //sort via date
            Console.WriteLine("[date desc]");
            foreach (dynamic win in windows.OrderByDescending(x => ((IDictionary<string, object>)x)["released"]))
            {
                Console.WriteLine(win.windows + " - Codename:" + win.codename + " - Released: " + ((DateTime)win.released).ToShortDateString());
            }

            Console.ReadLine();
        }
    }
}

Accessing the Cloudflare API in C#

January 17, 2017 chrisbitting6 Comments

shutterstock_328592903 Cloudflare provides security, CDN and more for your websites. If you’re using the Cloudflare caching to speed up your sites (it really is fast) you may want to purge their cache from your application (instead of waiting X days). Cloudflare provides an API that seems to offer everything you’d possible need. I wanted to do this from c#, but did’t find any great libraries or code that was using their newest API (v4).

Below is some quick simple code that I’ve found to work great so far for me. It’s pretty basic, and doesn’t require many external libraries (just JSON.net). Let me know your thoughts and if it helps you.

//define somethings we'll need for the api
string apiEndpoint = "https://api.cloudflare.com/client/v4";

//user info here
string userEmail = "youremail@something.com";

//this is your Global api key found in "my account"
string userAPIkey = "xxxxxxxxxxxxxxxxxxxxxxxxx";

//domain your working with:
string domain = "http://www.yourdomain.com";

//let's get our zone ID (we'll need this for other requests
HttpWebRequest request = WebRequest.CreateHttp(apiEndpoint + "/zones?name=" + domain + "/&status=active&page=1&per_page=20&order=status&direction=desc&match=all");
request.Method = "Get";
request.ContentType = "application/json";
request.Headers.Add("X-Auth-Email", userEmail);
request.Headers.Add("X-Auth-Key", userAPIkey);

string srZoneResult = "";
using (WebResponse response = request.GetResponse())
using (var streamReader = new StreamReader(response.GetResponseStream()))

	srZoneResult = (streamReader.ReadToEnd());

dynamic zoneResult = JsonConvert.DeserializeObject(srZoneResult);

if (zoneResult.result != null)
{
	//get our zoneID
	string zoneId = zoneResult.result[0].id;

	//some pages to purge the cache on:
	byte[] data = Encoding.ASCII.GetBytes(@"{""files"":[""http://www.yourdomain.com/about/""]}");

	request = WebRequest.CreateHttp(apiEndpoint + "/zones/" + zoneId + "/purge_cache");
	request.Method = "DELETE";
	request.ContentType = "application/json";
	request.ContentLength = data.Length;

	request.Headers.Add("X-Auth-Email", userEmail);
	request.Headers.Add("X-Auth-Key", userAPIkey);

	using (Stream outStream = request.GetRequestStream())
	{
		outStream.Write(data, 0, data.Length);
		outStream.Flush();
	}

	string srPurgeResult = "";
	using (WebResponse response = request.GetResponse())
	using (var streamReader = new StreamReader(response.GetResponseStream()))

		srPurgeResult = (streamReader.ReadToEnd());

	dynamic purgeResult = JsonConvert.DeserializeObject(srPurgeResult);

	//was it a success (hopefully it = true)
	textBox1.Text = purgeResult.success;
}

Generating Random Passwords (with required characters) in c#

May 5, 2016May 10, 2016 chrisbittingpassword, randomLeave a comment

In looking for a way to create some passwords that included sets of characters (lower case, upper, numbers or symbols) below is a chunk of code that seems to do the job. It makes use of RNGCryptoServiceProvider from System.Security.Cryptography instead of the classic Random.

You can choose a set length or variable:

//use a set length
string pass = GenerateAPassword(10);

//or get a random length between two ints
string pass = GenerateAPassword(12, 15);

The required parts of the password are created using a simple string array. -Be careful is creating small sets of strings, as it could take longer to create a password to fulfill the requirements.

 string[] parts = new string[] { "abcdefghjkmnpqrstuvwxyz", "ABCDEFGHJKLMNPQRSTUVWXYZ", "23456789", "*&^%$#@!" };

Below is the needed code.

You’ll need: using System.Security.Cryptography & using System.Text at the least.

public string GenerateAPassword(int length, int maxl = 0)
        {
            //separate required parts of password below
            //below i have lower, upper, numbers and symbols
            string[] parts = new string[] { "abcdefghjkmnpqrstuvwxyz", "ABCDEFGHJKLMNPQRSTUVWXYZ", "23456789", "*&^%$#@!" };

            if (length < parts.Length + 1)
            {
                return "invalid length";
            }

            //
            if (maxl > 0)
            {
                length = rngNumber(length, maxl);
            }

            StringBuilder pass = new StringBuilder();
            int l = length;
            while (0 < l--)
            {
                pass.Append(string.Join("", parts)[rngNumber(0, string.Join("", parts).Length - 1)]);
            }

            //check of parts of password exist
            foreach (string part in parts)
            {
                if (pass.ToString().IndexOfAny(part.ToCharArray()) == -1)
                {
                    //create again if missing
                    return GenerateAPassword(length);
                }
            }

            return pass.ToString();
        }

        private static readonly RNGCryptoServiceProvider rng = new RNGCryptoServiceProvider();

        public static int rngNumber(int min, int max)
        {
            byte[] randomNumber = new byte[1];

            rng.GetBytes(randomNumber);

            return (int)(min + (Math.Floor(Math.Max(0, ((Convert.ToDouble(randomNumber[0])) / 255d) - 0.00000000001d) * (max - min + 1))));
        }

DataTable Row Loop c# Performance Testing (Linq vs. Select vs. Parallel vs. For)

May 4, 2016May 10, 2016 chrisbitting.net, datatable, enumeration, loop, performance, stopwatchLeave a comment

I still find myself using DataTables (from SQL Server, mySQL, etc.) on a regular basis. In an effort to see how some different methods of looping through the data performs and what method might be the fastest, I put together some small, fast tests, below are my test results.

[Just jump to the fastest method.]

My sample data was comprised of about a 34,000 row datatable – running these tests on a i7, 32gb, VS2015 setup. The 34,000 was chosen because a box of Froot Loops contains about 1800 Froot Loops (I think), but 1800 was too small, so how about 18 boxes of Froot Loops?

Continue reading “DataTable Row Loop c# Performance Testing (Linq vs. Select vs. Parallel vs. For)” →

Parsing JSON data in C# (JSON.NET, Linq, HttpClient)

May 2, 2016 chrisbittinghttpclient, json, linqLeave a comment

Parsing JSON in C# is pretty simple. I’m a fan of using Newtonsoft.Json (known as JSON.NET) – I’ve found this to be the fastest, easiest JSON parser available.

Below are some steps to get parsing on your own (or jump down to the full source code). You can also grab this on GitHub. I’m just using a simple console app for this example. My JSON is also coming from a Google Custom Search (via http get). You’ll need to have your own JSON url (or maybe get a test url here).

Start a new project > console application.
In Package Manager Console, run: Install-Package Newtonsoft.Json
Add the below to the top of your code file (Program.cs):
using Newtonsoft.Json.Linq;
using System;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Threading.Tasks;
Create a task to get some JSON data from http:public static async Task<string> httpResponse(string url)
{
IWebProxy iprox = WebProxy.GetDefaultProxy();
iprox.Credentials = CredentialCache.DefaultCredentials;HttpClientHandler httpHandler = new HttpClientHandler()
{
UseProxy = true,
Proxy = iprox,
PreAuthenticate = true,
UseDefaultCredentials = true,
Credentials = CredentialCache.DefaultCredentials
};using (var httpClient = new HttpClient(httpHandler))
return await httpClient.GetStringAsync(url);
}
Now get some data:
JObject jData = JObject.Parse(httpResponse(@”https://www.googleapis.com/customsearch/v1?youneedtocreateyourownlink”).Result);
Read some simple data:
string sampleValue = (string)jData[“url”][“type”];
Read array of data:
foreach (var sampleItem in jData[“items”])
{
string sampleItemValue = (string)sampleItem[“link”];
Console.WriteLine(sampleItemValue);
}

The full source:

using Newtonsoft.Json.Linq;
using System;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Threading.Tasks;

namespace parsingJSON
{
    internal class Program
    {
        public static async Task<string> httpResponse(string url)
        {
            //i'm using a proxy, you could remove this if needed.
            IWebProxy iprox = WebProxy.GetDefaultProxy();
            iprox.Credentials = CredentialCache.DefaultCredentials;

            HttpClientHandler httpHandler = new HttpClientHandler()
            {
                UseProxy = true,
                Proxy = iprox,
                PreAuthenticate = true,
                UseDefaultCredentials = true,
                Credentials = CredentialCache.DefaultCredentials
            };

            using (var httpClient = new HttpClient(httpHandler))
                return await httpClient.GetStringAsync(url);
        }

        private static void Main(string[] args)
        {
            //get some JSON data
            JObject jData = JObject.Parse(httpResponse(@"https://yourJSONlinkhere").Result);

            string sampleValue = (string)jData["url"]["type"];

            Console.WriteLine(sampleValue);

            //loop through array in JSON
            foreach (var sampleItem in jData["items"])
            {
                string sampleItemValue = (string)sampleItem["link"];
                Console.WriteLine(sampleItemValue);
            }

            //filter w/ linq:

            var sampleArray =
              from p in jData["items"]
              select p;

            foreach (var sampleItem in sampleArray.Where(p => ((string)p["link"]).Contains("https")))
            {
                string sampleItemValue = (string)sampleItem["link"];
                Console.WriteLine(sampleItemValue);
            }

            Console.ReadLine();
        }
    }
}

View on GitHub

Parsing XML with HTML Agility Pack (instead of XDocument, etc.)

April 15, 2016April 15, 2016 chrisbitting2 Comments

If you’re looking to easily parse some XML w/ .net and finding the normal XObject (Linq) isn’t cutting it, using HtmlAgilityPack could be your answer.

Below I’ll layout the simple steps for parsing a XML file w/ HtmlAgilityPack. The XML file I’m using to test is the WordPress export XML format. It uses lots of XML features so seemed like a good test. Let’s start. I’m using Visual Studio and in my example I’m assuming you have a project already open (I’m using a simple Console app). Continue reading “Parsing XML with HTML Agility Pack (instead of XDocument, etc.)” →

Chris Bitting

Random, I know.

.net