Category: Web Development

Posted on

User Control over the Granularity of Location Services

I use a lot of location services on my phone: when I tweet, more often than not, I include my location. I love geo-tagging my photos, so I generally include my location when using instagram, Hipstamatic, etc. & I regularly check in on Gowalla & Foursquare. So I’m not averse to sharing my location in general. I actually quite like it. That being said, I often wish I could be less specific about where I am. I don’t think it would be too hard to add a little slider, or some interface, to provide some scale.

By default, we send data for a point. But what if I could choose to send data at a variety of scales: point, neighborhood, city, region, province/state.

I suppose the particular use-case for this to avoid sending the exact location of my house – I do it, somewhat inadvertently, but I could imagine not ever wanting to do it. But still, letting people know that I am currently in South Vancouver (neighborhood), or Vancouver (city), or Lower Mainland (region), or BC (province/state), rather than my location within 100 metres should be perfectly acceptable data points – and gives me some control over the specificity of my data points.

In the above example, it is up to the app developer to provide scale/fudge-factor options. But we could abstract this farther, and make it a device-wide setting. My phone, via GPS, can always tell where I am. What if I could, device-wide, say “When in these areas, broadcast my location with much less specificity. That way, when I’m actually at home, it could automatically just send, say “Vancouver”, rather than my location. And by letting me choose where I want to reduce specificity, I still have the control – I set it up in my settings or preferences.

I suspect there’s a variety of implementation details that I haven’t really thought through, but I do think that this is an issue that if not the device (/OS) makers need to address, than app-developers do. Let me participate in location services, but at my security level – not what you’d ideally want. It’s a users-first approach to location, rather than data-first.

Posted on

Optimizing site speed: a case study

One of our clients runs an eCommerce site selling workout videos on behalf of George St.Pierre, the UFC Fighter, called GSPRushfit. The videos sell well, but they’re quite rightly always looking at ways to increase sales. A few weeks ago, we ran a study to see how fast the site loaded, and how that affected conversion rates (sales). I wrote a post about how we measure that a couple of weeks ago. The result of this was that we could see that page loaded reasonably well, but not fantastically. Across all pages, the average load speed was 3.2 seconds. What was eye-opening was that pages that loaded in 1.5 seconds or less converted at about twice the rate of pages loading 1.5-5 seconds. There was  a further dip between 5-10 seconds. So with this data in-hand, I started to look for ways to increase page load speed. I came up with a laundry list of things to do. Most of these are suggested by YSlow:

  1. Remove inline JS/CSS: We didn’t have a lot, but there was some unnecessary inline scripting. These were moved into the existing CSS & JS files. I think I added about 50 lines of code. Not a lot, but helpful. There’s still some inline JS & CSS that’s being written dynamically by coldfusion, but all the ‘static’ code was moved into one.
  2. Minify Files: This doesn’t do a lot, but does compress files slightly. I think I was able to reduce our JS file by 30% & our CSS by 15%. Not a lot, but still helpful. I use an app called Smaller, which I’m a big fan of. While YSlow suggests you combine files, I chose not to – the reduction in requests didn’t offset the problems for us in doing this.
  3. Reduce Image Size. The site is fairly graphically intensive – large background images & lots of alpha-transparency PNGs. When I started, the homepage was loading just under 1.2MB in images, either as CSS backgrounds or inline. Without (to my eye) any noticeable loss of quality I was able to re-cut those same images to about 700KB in size.
  4. Use a CDN: The site loads video, which we call from a CDN. But the static assets (CSS, JS, Images) weren’t being pulled. This was originally because the CDN doesn’t support being called over SSL. But it only took a little scripting to have every image load from the CDN while not on SSL, from the local server while over SSL. This, as you’d expect, greatly improved the response time – by about 0.2 seconds on average.
  5. Query Caching: this one is likely a no-brainer, but the effects were stunning. All the content is query-driven, generated by our CMS. But it doesn’t change terribly often. So I started caching all the queries. This alone dropped our page load time by nearly a full second on some pages. And to maintain the usefulness of a CMS, I wrote an update to clear specific queries in the cache when new content was published.
  6. GZip: Again, likely something I should have already been doing, but to be honest, I had no idea how to accomplish this on IIS. So I figured that out and requested that the server gzip static assets (JS, CSS & HTML files).
  7. Far-future expires headers. Because very few of our images change frequently, I set an expiry date of 1 year in the future. I likewise set a cache-control variable of the same time frame. Which, should, in theory, reduce requests and allow clients revisiting to just use their local cache more. I of course added a programmatic way to clear that cache as well for when we change content or edit the JS or whatever.
  8. Clean up markup: While I was doing all the rest, I also cleaned up the markup somewhat – not a lot, but again, we were aiming to eliminate every extraneous byte.

So, as you can see, we sort of threw the kitchen sink at it to see what stuck. In retrospect, I wish I had made these updates one at a time to measure what sort of an impact (if any) each one had. There’s only a couple I can see clear before & after differences, which were mentioned above. So for everyone out there, which of these were the most effective?

  1. Caching Dynamic Content: Even on a CMS-driven site, most content doesn’t change constantly. But if you can eliminate trips to the DB server to make a call, that’s time saved. Even if you cache a recordset for a few minutes or even just a few seconds, on a high-traffic site you can see some real impressive gains. We cache queries on this site for 10 days – but can selectively update specific queries if a user makes an edit in the CMS – sort of a best of both worlds right now. This somewhat depends on having a powerful server – but hosting hardware & memory are pretty cheap these days. There’s no reason not to make use of it.
  2. Crushing images: When building the site, I did my best to optimize file size as I exported out of Photoshop. but with a few hours in Fireworks I was able to essentially cut the size of the images in half with no real visual impact. A hat-tip to Dave Shea for the suggestion of using Fireworks.
  3. Pushing Content to a CDN: this is the head-smacking no-brainer that I don’t know why wasn’t already part of our standard workflow on all sites. As I wrote above, we gained about 0.2 seconds by doing this – which doesn’t sound like a lot, but it’s noticeable in practice.

The nice thing about this exercise was that it shook up how I built sites, how our internal CMS runs and how I configure both our IIS and Apache servers to all run slightly more efficiently. I suspect that I could eke out a few more milliseconds by playing more with the server settings itself, but I’m satisfied for now with how this worked out.

Posted on

Measuring Page Load speed with Google Analytics

UPDATE: Google now has a built-in tool for doing this. Simply add the following: _gaq.push([‘_trackPageLoadTime’]); and hey presto: Page-load-speed tracking. See this page for more info, including an important caveat.

With Google now including Page Load speed in their page-ranking algorithm, many of our clients started asking us how fast their sites load. There’s lots of developer tools that can help with this – I use (alternately) the Google Page Speed tool and Yahoo! YSlow during development to tweak page load times. But this doesn’t help our clients very much.

There’s a chart you can use on Google Webmaster Tools, but overall, I don’t find Webmaster Tools particularly end-user friendly. As it’s name implies, I’m not sure that it is supposed to be. A couple of our more technical clients use it, but most don’t use it, or don’t really get it. The chart that Google Webmaster Tools spits out can be found under “Labs”, then “Site Performance”. It looks like this:

Google Webmaster Tools Page Speed Chart
Google Webmaster Tools Page Speed Chart

Which is nice, and gives a clear delineation of what’s good, and what’s not. As you can see, this site rides right on the edge of being “fast”. When a client asked if I could tell him how fast individual pages are loading, this gave me the idea of using Google Analytics’ Event Tracking feature to log how fast a page loads. What’s nice about this is that most of our clients “get” Analytics, and are used to going there to see their stats. So additional stats there works for them.

With Google’s visual guidelines in place, I set about making this happen. I decided to name the Event Category “PageLoad”. I then added 5 labels to group the results:

  1. Fast (less than 1500 MS) (because Google, according to Webmaster Tools, considers anything 1.5s or faster “fast”.
  2. Acceptable (less than 3000 MS)
  3. Middling (less than 5000 MS)
  4. Slow (less than 10000 MS)
  5. Unacceptable ( 10000 MS or longer)

These groupings are completely arbitrary – I could have set them at any time span. Those just seemed reasonable to me, based on knowing that most of our sites have an average load time of 2800 MS, based on our internal tools.

So then I had to track it. The code is as follows:

var pageLoadStart = new Date();

window.onload = function() {
var pageLoadEnd = new Date();
var pageLoadTime = pageLoadEnd.getTime() - pageLoadStart.getTime();
// let's set some second (1000s of ms) segments
if (pageLoadTime < 1500)
    loadStatus = 'Fast (less than 1500 ms)';
else if (pageLoadTime < 3000)
    loadStatus = 'Acceptable (less than 3000 ms)';
else if (pageLoadTime < 5000)
    loadStatus = 'Middling (less than 5000 ms)';
else if (pageLoadTime < 10000)
    loadStatus = 'Slow (less than 10000 ms)';
else
    loadStatus = 'Too Slow (more than 10000 ms)';
var myPath = document.location.pathname;
if( document.location.search)
    myPath += document.location.search;
// round the time to the nearest 10 ms.
pageLoadTime = Math.round(pageLoadTime / 10) * 10;
// send the GA event
try {
    _gaq.push(['_trackEvent', 'Page Load', loadStatus,myPath,pageLoadTime]);
} catch(err) {}
}

Some Notes:

  • I have the first line (var pageLoadStart = new Date();) at the very top of my header, right after the <head> tag. The window.onload() function sits in the footer.
  • I round my values to the nearest 10 MS – but this might be too many values, too noisy. So you could round to the nearest 100 MS or even nearest second if that worked  better for you.
  • the document.location.pathname just passes the page’s address to Google, so I can see which pages are slow, or when particular pages or slow.
  • You can only send integers to the Event Tracking widget.
  • the _gaq.push() line is where I send this to Analytics. You might have your Analytics tide to a different variable, in which case change the _gaq to whatever you use. For more on how event tracking works, see the docs

What my client then sees in Google analytics is a chart like this (note – I’m using the new version of Analytics – yours might look different):

 

PageLoad chart
PageLoad chart

As you can see (this is for the home page of the site), the results are ALL over the place. Which is troubling, but that’s for another day to figure out why it sometimes loads sooooo sloooowwwwlyyyyy. (We’re currently running an A/B test wherein a video auto-plays or not. My best guess is that because the page doesn’t finish loading until after the video starts to play, that explains the slow loading. But maybe not).

Regardless of the actual performance of this particular page, you can see how this is a nice little chart for clients to see – nothing too technical (I could possibly be even less technical by not including the MS numbers in the labels), but provides them some easy insight into how their site is doing.

Caveat: This chart obviously doesn’t include any pre-processing factors – connection speed, server-side scripts taking a long time, etc. But it’s still pretty useful, I think.

Posted on

Coldfusion Java.lang.ThreadDeath

Of  late, we’ve stared seeing a raft of errors of type “java.lang.theadDeath” on our Coldfusion-based servers. In every case, these errors were generated by the CFSchedule User Agent, meaning that scheduled tasks were triggering them. Of more interest, they were, with 1 exception, being thrown by a scheduled CFPOP call to handle email newsletter bounce-backs by our eNewsletter delivery system. The scary part is that, more often than not, our server would start to hang shortly after getting a bunch of these and we’d have to restart the service. Not ideal on servers that require very uptime on account of some exceedingly busy eCommerce sites.

After casting about fruitlessly on Google & Bing for some answers to this, I buckled down to investigate the errors more closely myself. And I believe I have figured it out:

Each of our websites has a Cron handler: We add in any number of scripts we wish to call, and interval between & whether it is allowed to run concurrently with anything or not. The central ColdFusion scheduler service then simply calls this handler for each site at regular intervals (generally, we call the handler about every minute. The tasks themselves range from every 2 minutes to monthly). This handler runs multi-threaded – that is to say that, if a task is allowed to run concurrently with other tasks, it’ll spawn a thread for each task, run it, then tie up the threads afterwards.

So here’s where I think this issue arrives from: According to the very scant documentation and blogs I could find, this error-type will be handed off when a thread is killed off by the server after being queued for too long, awaiting execution, or for the loser in race conditions. We had our server to allow for 10 concurrent threads. Which is pretty small, but seemed ok. My guess is the connection to the mail server when popping messages runs threaded itself. So we were spawning massive number of threads. Further to that, our allowed timeout on mail server connections was longer than our allowed request time or thread-queue time. So threads were constantly timing out, or being killed off because they were queued for too long – which would then spit that error.

Given that we have oodles of RAM, I’ve both upped the number of concurrent threads we allow, as well as reduced the mail-server connection timeout to be less than both our request-timeout and queue-timeout. Testing throughout today appears to have solved this, but I’ll have to watch & see over the next few if indeed this is now solved.

Posted on

Calendar Synch

At work, we use a ticketing/timetracking app that I wrote a while back. It’s served us well, and, hopefully in the near future, we’ll roll it out to be a full-fledged stand-alone product (we’re doing some closed beta-testing right now – if you’re a consultancy/digital studio looking for something like this, let me know!).

Today I finally added a feature I’ve been wanting for ages: Synching the ticket-list with my calendars. It’s a fairly simple process. Based on the query string, I can spit out a webcal:// feed of all tickets, all tickets for a particular client, tickets of a particular type, assigned to a particular person, in a particular state, or any combination of any of those. These then just show up in my calendar under my Calendar subscriptions, and allow me to see at a glance what’s coming up. I’ve set my subscription to refresh hourly so I can see what everyone’s doing.

Currently, I’m only writing iCal events, but I’m playing with being able to send to-do lists this way. For “tasks” that are meetings, I can send these as invites to the attendees, which is pretty cool too – all through our centralized ticketing & time-tracking system. Mostly, as a manager, I can quickly look at the calendars for my staff to see who’s working on what when so I have a quick idea of availability. And even my own, which I sometimes lose track of.

Writing an ICS file turns out to be even more trivial that I had thought it would be – I honestly don’t know why I don’t have this as a standard part of every calendar module on every site as a subscription option. Now that I’ve written a simple function to pass in an array of date info and spit out an ICS file, I likely will.

But what’s most rewarding about this work is the immediate use this has received by our beta-test group – who were used to using calendars to manage their work before, but wanted something more central – so now they have the best of both worlds: centralized, web-based ticketing/time-tracking, but each user can still continue to use their desktop & phone calendars as they used to.

It’s nice when an afternoon’s project works out so well!

Posted on

A Day Apart Seattle – my thoughts on the workshop

Last week, I posted my thoughts on Day 1 of An Even Apart  (Seattle). I had fully planned on writing on both day 2 & the 3rd-day workshop, but after somehow losing a longish post to the ether, am skipping day 2. Suffice to say, the quality of the talks continued, as well as the laser-like focus on what can be accomplished with HTML5 and CSS3.

The A Day Apart Workshop was my primary reason for attending the conference. I wanted some hands-on learning & experimentation with these new tools that I had been unable to play with much. I have seen both Jeremy Keith & Dan Cederholm speak previously, so knew they would be good. And they were good. In fact, I would argue that the only thing that made this workshop worthwhile in the end was the quality of the presenters, who overcame everything to actually deliver quality.

This was the first A Day Apart put on by the AEA folks, so I expected it to not be perfect. But I did expect more. I’m certain that the next iteration will improve slightly on this one, and so on and so forth. But let me list my complaints, with some hopefully helpful critique:

  • Size of workshop: There were simply too many attendees in one room. It meant, for the most part, that it was hard to carry on a Q&A thread, because you don’t want to take up everyone else’s time & B, radically changed the possibilities of how run the workshop. To fix, I’d recommend a)lessening the number of spaces available and b)split the audience into 2 groups. Group A would get presenter #1 in the AM, presenter #2 in the PM. Group B would get it in reverse. Yes, this would double the work-load for the speakers – but I think would vastly enhance the experience for the attendees.
  • Format of workshop: The workshops were really just extended seminars – not so much a workshop.  While Jeremy’s seminar & workshop topics were different, Dan was talking about CSS3 in both, so his workshop felt like an extension of his earlier seminar. Jeremy had us guess the definitions of elements from the HTML 5 spec, and actually work that out. The result? I remember those definitions more than virtually anything else. Because I got to actually interact with the material. So here’s my suggestion to fix this: Each workshop was divided into 3 parts (if I recall correctly). If, let’s say, 10 minutes was cut out each (maybe even 20 minutes) and replace with a related exercise for the audience to do, suddenly the interaction with the material would increase greatly, and, I suspect, both people’s comprehension & retention of the material. If, in addition to 10 minutes of homework, there was schedule in 10 minutes for post-homework Q&A, that provides a nice way to summarize each content block.
  • Density of Material: The material-to-time ratio was way off, which meant we raced through the material. Either cover less or make the workshop longer. Both spent a long time on the history of the material – this was useful, but could likely have been done quicker, given the quality of the rest of the meat they were delivering.
  • Related Assets/sample code: We were all given a book with all the slides printed and bound into it. This is a great reference. But an online wiki, perhaps specific to the course, that was setup by the creators, but, going forward, be looked at, edited, updated by the attendees would be awesome. If a laptop was required (given the audience, I don’t think that’s an unreasonable request) and we were provided links to download some source code, we could then, in the workshop, very quickly build a site each, to see the material in action ourselves – this would work great with the ’10 minutes of homework per section’ model – you just keep building on. This has been, more or less, the standard for programming workshops I’ve attended. For me, this is likely the biggest miss.

Despite everything I’ve written above, I do feel that I came out that workshop with a better understanding of the two tools I went into the workshop wanting to learn. As I said above, this is almost entirely due to the quality of the speakers, not the format of the workshop itself. If there was to be another A Day Apart at a future conference, on a topic of interest to me, I would certainly consider attending it again. But while it was OK, it could have been great.

Update (2010-4-12): I had initially remarked that both workshops felt like extensions of the seminar. Jeremy, thankfully, corrected me that his seminar and workshops were quite different. My apologies.

Posted on

An Event Apart Seattle (2010) – Day 1

I spent the first 3 days this week at An Event Apart (Seattle). This is a conference that I’d been wanting to attend since its inception, but somehow never actually made it down to one. I was really looking forward to a few days of self-affirming web-geekery. And that respect, I wasn’t disappointed.

I don’t know the process by which An Event Apart selects their speakers, but whatever it is, it is good. From start to finish, the quality of the presentations were excellent – even those whose content I wasn’t particularly interested in. Jeffrey Zeldman got us started in fine form, with a talk, essentially, about mistakes that would be good to avoid in running a studio. Having been running a shop (or studio) for the past 7 years, this was of very little interest to me – I’ve made those same mistakes previously, I’ve come to many of the same conclusions, I’d offer the same advice to anyone wanting to strike out on their own. But! I still thoroughly enjoyed it. A light hors d’oeuvre before the meaty sessions that followed. He’s a great speaker, which made this otherwise too-low-level talk appreciable by all.

Of all the talks, Nicole Sullivan‘s, who followed next, was the least inspiring. It fell between two worlds for me. She was talking about object-oriented CSS. Given her background, I was hoping for a super-nerdy, intense look at site-speed optimization & whatnot. We got a little bit of that – but not with the detail I’d like, and then the second-half of her talk was spent looking at her wish-list for things to be included in future CSS spec. So, not even actual proposed spec. Things that she proposes should be in proposed spec that I might get to use in bleeding-edge browsers 3-4 years from now and actual projects a decade or so in the future. Which felt like a waste of my time, to be honest. So, while I’m down on her talk, her answers in the brief Q&A were great and I’d love to hear her do a “developer” talk, rather than a “designer” talk, which this seemed to be.

Dan Cederholm talked CSS3, and gave some nice tips & tricks. His presentations are fantastic – but I’ll talk more about him later. Luke Wroblweski gave the talk that I wish every designer in the world could hear. Titled “Mobile First!”, I think Jeffrey Zelman summed it up best: “Luke Wroblewski’s extraordinary “Mobile First” presentation changed the way I think about web design”. It was compelling, well-backed-up with samples, and, perhaps best of all, seemed very easy to implement.

Aaron Walter‘s talk ‘Learning To Love Humans—Emotional Interface Design’ was funny, humble  and very very smart – all about how to create an emotional response to design, and moving beyond the idea that functional is the goal (paraphrasing Aaron to sum it up: You never hear a chef say ‘taste this, it’s edible!’ so why should a designer).

If you’ve never heard Jared Spool talk about usability, design & process, chances are you’re doing it wrong. His insights are incredible. His talk here was about the anatomy of a design decision – what ‘kind’ of design to teams do, how they arrive at that process, and what effect it has both on productivity and on end-user experience. The 101-take: Experience is what happens in the space between actions. His talk, to me, nicely summed the internal conflict that makes Pencilneck Software work so well. I am, by default, an intuitive developer. I rely on tips, tricks, experience and instinct to guide me through what I do. Jeff, by contrast, is a firm believer in process & methodology to get things done right. Where we meet in the middle is why we are successful where lots of other firms have failed, I feel – and Jared Spool really captured both the differences in approach and how they each affect teams & workflow.

Posted on

Wanted: A twitter/comment plugin combo

When I post an entry (such as this one), I have a wordpress plugin (called Twitter Tools) that sends out a tweet (meta-linking update: like this) telling everyone that I’ve posted something. To manage comments, I use Disqus, which, amongst other systems, allows people to authenticate at twitter to then post a comment. Which is nice, and I like it.

But! Sometimes, seemingly more often than not, people will @-reply to me on twitter with a comment on my post. And I will often @-reply someone else about their post (that was announced on twitter). So here’s what I want:

  1. When I post an entry that sends a tweet, capture and store the ID of the tweet that I sent.
  2. Whenever someone @-replies or retweets that stored tweet, aggregate that to the comment-section of my blog, so that the entire related conversation is visible in one place.
  3. For bonus points, given that everything I post is also pushed as a post to Facebook & everything I tweet ends up as a status update on there too, it would be great to extract any responses to those as well, in the comments section of my site.

Does anyone know if such a thing exists?

Posted on

FCKeditor & Firefox 3.6

With the release of Firefox 3.6, those of you who interact with FCKeditor will note that it ceases to work, giving you instead a textarea. This is because in the browser sniffer, FCK ignores all browsers made in 2010 or later – such as Firefox 3.6. I can’t speak for any other versions, but in the Cold Fusion version, there’s a really simple fix:

  1. Locate fckutils.cfm, located in the root folder in the install.
  2. On line 47, edit the line that reads
    stResult = reFind( “gecko/(200[3-9][0-1][0-9][0-3][0-9])”, sAgent, 1, true );
  3. Edit the reFind to read:
    stResult = reFind( “gecko/(20[0-9][0-9][0-1][0-9][0-3][0-9])”, sAgent, 1, true );

This will give you another 90 years. Which should be plenty of time to find a replacement, particularly as the new CKeditor is now out, and doesn’t have this issue.

Posted on

Newsletter Stats from the Pencilneck CMS

So our CMS, the Pencilneck CMS includes a email newsletter management system. With a little time between finishing one project & starting the next, I took some time to analyze the data we generate – in part out of self-interest, but also as part of a new project to provide additional analytics tools for our clients. Having spent the better part of a morning generating data, I thought I’d share this information with the general public. I’d love to know how this corresponds to the data from other newsletter delivery systems.

All times report are Pacific Time. All hours cited include the entirety of the hour. For instnace, 8AM means all time between 8:00:00 and 8:59:59, inclusive. For time-of-day reporting, I divided up the hours into 4 blocks – unequal in length but useful for business hours:

  • Midnight-7AM: the Pre-work period
  • 8AM – 11AM: the morning at work.
  • Noon-4PM: the afternoon at work.
  • 5pm-11PMt: the evening at home

Number of Newsletter Recipients: 7,956,301

These first few numbers are based on the number of sends, so when our clients decided to send a newsletter.

Most Popular Day to Send a Newsletter: Tuesday (34%), Monday (28%).
Least Popular Day to Send a Newsletter: Saturday(1%), Friday (5%).

Most Popular Time of Day to Send a Newsletter: 8AM-11AM (40%), Midnight-7AM (25%)
Least Popular Time of Day to Send a Newsletter: 5PM-11PM (2%),Noon-4PM(10%)

Most Popular Day & Time to Send: Tuesday Midnight-7AM (34%), Monday 8AM-11AM (30%)
Least Popular Day & Time to Send: Friday 5PM-11PM(1%), Saturday 5PM-11PM (3%)

These next set of numbers are the “best” results based on open rate. The Open rate was calculated by dividing the number of newsletters that were opened vs the number of newsletters received. A newsletter is considered received if our system did not receive any bounce notifications regarding the delivery.

Best Day to Send a newsletter: Sunday (35%), Monday (33%)
Worst Day to Send a newsletter: Saturday (2%), Friday (8%)

Best Time of Day to send a newsletter: Midnight-7AM (40%), 5PM-11PM(35%)
Worst Time of Day to send a newsletter: 8AM-11AM (5%), Noon-4PM (8%)

Best Day & Time to send a newsletter: Monday Midnight-7AM (65%), Sunday 5PM-11PM(45%)
Worst Day & Time to send a newsletter: Saturday Midnight-7AM(1%),Friday Noon-4PM(3%). And worth mention is Wednesday Noon-4PM (4%).

We also, by way of Geolocation via IP address, can track where a recipient opens a Newsletter. For our clients, we group this by Province & State for Canada, US, Mexico & Australia, and the rest of the world simply by Country. I discounted all geo-data with less than 100 total recipients, to single out places like Ghana & its 100% open rate based on 4 recipients. Actually, the same recipient, 4 times.

Best open rate, by Recipient location: Ontario (65%), British Columbia (45%)
Worst open rate, by recipient location: Alberta (4%), Minnesota (5%)

Finally, I looked at links, and where they appear within the source code. To do this, I stripped all the header & styling information from the HTML source, leaving just the body, and divided that into quarters to see which links were the most clicked on. These results are probably not that surprising, but nevertheless. I determined the click rate by calculating the total number of recipients who clicked on a link vs the total number of opened newsletters. I then split that up based on which portion of the newsletter the click was in.

Overall clickthrough rate: 11%.
Average number of links per newsletter: 6
Most links in a newsletter: 104

Clickthrough rate for links in the third quarter of content: 9%
Clickthrough rate for links in the first quarter of content: 19%.
Clickthrough rate for links in the second quarter of content: 8%
Clickthrough rate for links in the third quarter of content: 4%
Clickthrough rate for links in the last quarter of content: 15%

Given these numbers, I then calculated the overall percentage of links per quarter of content:

Links found in the first quarter of content: 14%
Links found in the second quarter of content: 46%
Links found in the third quarter of content: 28%
Links found in the last quarter of content: 12%

I’m not entirely sure what to make of all this. Some other random notes:

  • In general, Ontario & BC have a vastly higher open rate than anywhere else in Canada.
  • For the US, the 2 coasts open a rate about twice as much as middle America.
  • Germans are the most diligent European newsletter recipients.
  • It seems fairly clear that the best way to get someone to read your newsletter is to have it arrive before they get to their desk, but not so early as to have a lot of mail arrive between it arriving and the recipient checking their mail. Monday mornings are definitely the best for this.
  • Wednesday shows a general dipping in all stats. It really is hump day.
  • People skim newsletters: That’s why the top & bottom of your newsletter have the best click-through rates. This is likely partially explained by “Click here to view online” & Logos at the top & Unsubscribe links at the bottom of newsletters.
  • I’d love to do some signal-vs-noise analysis of newsletter – that, the clickthrough rates based on % of content that IS as link vs % of content that is NOT a link. My belief is that briefer newsletters do better.

I’ve stored all this data, along with the queries used to generate it all so I can now do this semi-regularly. Ideally, I’ll find some time to craft a simple API on this so that anyone could pull this anonymous, aggregate data to use, but I suspect that will be a long-term back-burner sort of project.

%d bloggers like this: