Spider / Download Website with Wget on Windows

Wget has been around forever it seems, but is still get great tool for spidering or downloading content (all content) from a website to your local machine. Below are some simple steps to install Wget and to run it.

To install Wget:

  1. Visit http://gnuwin32.sourceforge.net/packages/wget.htm and download the setup file, labeled “Complete package, except sources”.
  2. Run the setup .exe and leave everything as defaults.
  3. In a command prompt (cmd.exe) change to the C:\Program Files (x86)\GnuWin32\bin> directory (or C:\Program Files\GnuWin32\bin if your on a 32 bit OS)
  4. To test, if you just run wget.exe it should return wget: missing URL

 

To download a website:

  1. In a command prompt (cmd.exe) change to the C:\Program Files (x86)\GnuWin32\bin> directory (or C:\Program Files\GnuWin32\bin if your on a 32 bit OS)
  2. Now run: wget -e robots=off –no-check-certificate –recursive –level=0 -P “c:\\somefolder” http://chrisbitting.com (obviously replace my website with your own)

In that folder, you should soon files from the site Wget copied.

I’ve only had issues with a site that I had forced to use SSL. Wget didn’t like that.

Spider / Download Website with Wget on Windows

Using Amazon Polly from .net / c#, Get MP3 File

knight-rider-car-kittIf you haven’t checked out Amazon’s new Polly (Text-to-Speech (TTS) cloud service) it does produce some pretty great, life-like audio. Below is a quick sample on how you can feed some text to Polly, and get an MP3 file with it. I don’t cover all of the install options of the AWS Toolkit for Visual Studio / .Net, but it’s pretty simple. (listen to this text here)

Below is some code:

AmazonPollyClient pc = new AmazonPollyClient();

SynthesizeSpeechRequest sreq = new SynthesizeSpeechRequest();
sreq.Text = "Your Sample Text Here";
sreq.OutputFormat = OutputFormat.Mp3;
sreq.VoiceId = VoiceId.Amy;
SynthesizeSpeechResponse sres = pc.SynthesizeSpeech(sreq);

using (var fileStream = File.Create(@"c:\yourfile.mp3"))
{
sres.AudioStream.CopyTo(fileStream);
fileStream.Flush();
fileStream.Close();
}

 

Also make sure you have the below included:

using Amazon.Polly;
using Amazon.Polly.Model;

And these 2 NuGet packages added to your project:

aws-polly

This only scratches the surface of what Polly can do. Between streaming, SSML, Lexicons and more at a great price, I think we’ll be seeing more applications use this.

Using Amazon Polly from .net / c#, Get MP3 File

Google Chrome – Getting Entire List of Twitter Followers

The Bad:

Warning: this is an unofficial, non-Twitter approved way to easily gett a list of your Twitter followers.

The Good:

This is a really easy, can’t hurt anything method. Using just Google Chrome Developer Tools (part of the browser). But hey, if Twitter sends you a nasty letter, please don’t blame me.

The Scenario:

As of today, Twitter only shows about 18 of your followers at once. As you scroll down on your page, it loads another 18, so on and so on. If you have thousands of followers, getting to your first followers is kind of a pain, and would take forever. The below few chunks of javascript basically scroll the page to the end, and allow you to see the entire list.

Step 1. Open dev tools by hitting F12.

Step 2. Click the Console tab (maybe between Elements and Sources).

Step 3. Visit your followers page (ie: twitter.com/username/followers).

Step 4. Paste the below code into the Console (next to the “>”):

for (i = 0; i < 5000; i++) {

    setTimeout(function() {

        if ($(".has-more-items")[0]) {
        
            window.scrollTo(0, document.body.scrollHeight);

        } else {
           
           //console.dir('Finished.');
        }

    }, i * 1000);

}

Step 5. When it finished at and the bottom of your page, run the below to get a list of your followers copied to the clipboard:

var r = '';
$( ".ProfileCard-screenname .u-linkComplex-target" ).each(function( index ) {
r = r + $(this).text() +  "\n";
});

console.dir(r);
copy(r);

 

Bonus:
Do the above Step 4 on your “following” page, and then run the below to show people you’re following, but don’t follow you:

$( ".u-size1of2:has(.FollowStatus)" ).hide();
Google Chrome – Getting Entire List of Twitter Followers

SQL Server Date Convert Samples

Many times I want a particular date format but can’t remember which style it is. Below is a list of the styles and examples of how it will display. I don’t know if there are any additional Styles, if you know of any, comment and I can add them.

Usage:

CONVERT(VARCHAR, GETDATE(), 1) AS [Date]
Style Date
1 02/24/17
2 17.02.24
3 24/02/17
4 24.02.17
5 24-02-17
6 24 Feb 17
7 Feb 24, 17
8 13:10:36
9 Feb 24 2017 1:10:36:037PM
10 02-24-17
11 17/02/24
12 170224
13 24 Feb 2017 13:10:36:037
14 13:10:36:037
20 2017-02-24 13:10:36
21 2017-02-24 13:10:36.037
22 02/24/17 1:10:36 PM
23 2017-02-24
24 13:10:36
25 2017-02-24 13:10:36.037
100 Feb 24 2017 1:10PM
101 02/24/2017
102 2017.02.24
103 24/02/2017
104 24.02.2017
105 24-02-2017
106 24 Feb 2017
107 Feb 24, 2017
108 13:10:36
109 Feb 24 2017 1:10:36:037PM
110 02-24-2017
111 2017/02/24
112 20170224
113 24 Feb 2017 13:10:36:037
114 13:10:36:037
120 2017-02-24 13:10:36
121 2017-02-24 13:10:36.037
126 2017-02-24T13:10:36.037
127 2017-02-24T13:10:36.037
131 28/05/1438 1:10:36:037PM
SQL Server Date Convert Samples

Sorting ExpandoObject / Dynamic Object Lists in c#

Sometimes I find myself using lists of ExpandoObjects to quickly create lists of dynamic objects. They are super fast to create, and so flexible. You may however want to sort the list using one of the dynamic fields you added. Below is an easy way I use:

yourlist.OrderBy(x => ((IDictionary<string, object>)x)[“yourfield”])

Below is a complete example, making use of a couple Windows versions for some fun sample data.

using System;
using System.Collections.Generic;
using System.Dynamic;
using System.Linq;

namespace ExpandoObjectSort
{
    internal class Program
    {
        private static void Main(string[] args)
        {
            IList<ExpandoObject> windows = new List<ExpandoObject>();

            //add some data
            dynamic win31 = new ExpandoObject();
            win31.codename = "Janus";
            win31.windows = "Windows 3.1";
            win31.released = new DateTime(1992, 4, 6);
            windows.Insert(windows.Count, win31);

            dynamic win95 = new ExpandoObject();
            win95.codename = "Chicago";
            win95.windows = "Windows 95";
            win95.released = new DateTime(1995, 8, 24);
            windows.Insert(windows.Count, win95);

            dynamic winXP = new ExpandoObject();
            winXP.codename = "Whisler";
            winXP.windows = "Windows XP";
            winXP.released = new DateTime(2001, 10, 25);
            windows.Insert(windows.Count, winXP);

            dynamic win8 = new ExpandoObject();
            win8.codename = "Blue";
            win8.windows = "Windows 8";
            win8.released = new DateTime(2012, 10, 26);
            windows.Insert(windows.Count, win8);

            //loop through the list:
            Console.WriteLine("[default]");
            foreach (dynamic win in windows)
            {
                Console.WriteLine(win.windows + " - Codename:" + win.codename + " - Released: " + ((DateTime)win.released).ToShortDateString());
            }

            //sort via codename
            Console.WriteLine("[codename]");
            foreach (dynamic win in windows.OrderBy(x => ((IDictionary<string, object>)x)["codename"]))
            {
                Console.WriteLine(win.windows + " - Codename:" + win.codename + " - Released: " + ((DateTime)win.released).ToShortDateString());
            }

            //sort via date
            Console.WriteLine("[date desc]");
            foreach (dynamic win in windows.OrderByDescending(x => ((IDictionary<string, object>)x)["released"]))
            {
                Console.WriteLine(win.windows + " - Codename:" + win.codename + " - Released: " + ((DateTime)win.released).ToShortDateString());
            }

            Console.ReadLine();
        }
    }
}
Sorting ExpandoObject / Dynamic Object Lists in c#

Creating a Chrome Plugin to Scrape A Page (using jQuery)

If you’ve played w/ Chrome extensions at all, you know they are super powerful. I recently wanted to visit a bunch of pages, and extract some info from each page. I could easily run some jQuery script in the console of each page to do this, but I wanted a quick and easy way to do this. Creating a Chrome extension, that includes jQuery, to run locally is pretty simple. Below are the different files (5 of them) you’ll need (put these all in a single folder).

After creating these and adding your code, add to Chrome by going to Extensions > Load unpacked extension and choose your folder.

1: manifest.json

{
 "name": "Your Extension Name",
 "description": "This was easy",
 "version": "1.1",
 "background": {
 "scripts": [ "jquery-3.1.1.min.js","background.js","content.js"]
 },
 "permissions": [
 "tabs", "http://*/*", "https://*/*"
 ],
 "browser_action": {
 "default_title": "My Extension Title",
 "default_icon": "a-cool-logo.png"
 },
 "manifest_version": 1
}

2: jquery-3.1.1.min.js (get a copy from jquery.com)

 

3: a-cool-logo.png (16px x 16px)

 

4: background.js

chrome.browserAction.onClicked.addListener(function(tab) {
 chrome.extension.getBackgroundPage().console.log('your plugin gonna do something');
//maybe see if your plugin should be allowed to run
 if (tab.url.indexOf("/maybeCheckAurl/") !== -1) {
 
chrome.tabs.executeScript(null, { file: "jquery-3.1.1.min.js" }, function() {
 chrome.tabs.executeScript(null, { file: "content.js" });
});
 
 }
 
});

5: content.js (the magic happen here)

//i check to make sure jQuery is loaded
if (jQuery) { 
 
 jQuery(".someclass a").each( function() { 
 
//log the results 
 console.log($(this).attr("href"));

//maybe do something with them
 $.get( "http://yourapi"), function( data ) {});
 
 });
 
} else {
 alert('no jq');
}
Creating a Chrome Plugin to Scrape A Page (using jQuery)

Accessing the Cloudflare API in C#

shutterstock_328592903Cloudflare provides security, CDN and more for your websites. If you’re using the Cloudflare caching to speed up your sites (it really is fast) you may want to purge their cache from your application (instead of waiting X days). Cloudflare provides an API that seems to offer everything you’d possible need. I wanted to do this from c#, but did’t find any great libraries or code that was using their newest API (v4).

Below is some quick simple code that I’ve found to work great so far for me. It’s pretty basic, and doesn’t require many external libraries (just JSON.net). Let me know your thoughts and if it helps you.

//define somethings we'll need for the api
string apiEndpoint = "https://api.cloudflare.com/client/v4";

//user info here
string userEmail = "youremail@something.com";

//this is your Global api key found in "my account"
string userAPIkey = "xxxxxxxxxxxxxxxxxxxxxxxxx";

//domain your working with:
string domain = "http://www.yourdomain.com";

//let's get our zone ID (we'll need this for other requests
HttpWebRequest request = WebRequest.CreateHttp(apiEndpoint + "/zones?name=" + domain + "/&status=active&page=1&per_page=20&order=status&direction=desc&match=all");
request.Method = "Get";
request.ContentType = "application/json";
request.Headers.Add("X-Auth-Email", userEmail);
request.Headers.Add("X-Auth-Key", userAPIkey);

string srZoneResult = "";
using (WebResponse response = request.GetResponse())
using (var streamReader = new StreamReader(response.GetResponseStream()))

	srZoneResult = (streamReader.ReadToEnd());

dynamic zoneResult = JsonConvert.DeserializeObject(srZoneResult);

if (zoneResult.result != null)
{
	//get our zoneID
	string zoneId = zoneResult.result[0].id;

	//some pages to purge the cache on:
	byte[] data = Encoding.ASCII.GetBytes(@"{""files"":[""http://www.yourdomain.com/about/""]}");

	request = WebRequest.CreateHttp(apiEndpoint + "/zones/" + zoneId + "/purge_cache");
	request.Method = "DELETE";
	request.ContentType = "application/json";
	request.ContentLength = data.Length;

	request.Headers.Add("X-Auth-Email", userEmail);
	request.Headers.Add("X-Auth-Key", userAPIkey);

	using (Stream outStream = request.GetRequestStream())
	{
		outStream.Write(data, 0, data.Length);
		outStream.Flush();
	}

	string srPurgeResult = "";
	using (WebResponse response = request.GetResponse())
	using (var streamReader = new StreamReader(response.GetResponseStream()))

		srPurgeResult = (streamReader.ReadToEnd());

	dynamic purgeResult = JsonConvert.DeserializeObject(srPurgeResult);

	//was it a success (hopefully it = true)
	textBox1.Text = purgeResult.success;
}
Accessing the Cloudflare API in C#