Performance Matters

Index of Migrated Posts

2007-03-18T09:59:00.000Z

All posts from this site have now been migrated to their new locations at Web Performance Matters. This index is a cross-reference to the new site for anyone who arrives here from an old link.

My previous post (see below) also lists new locations for a few recent items that are not included in this list.

Performance Matters has moved

2007-02-22T22:15:00.000Z

I have joined UpRight Marketing as a partner, and after a lot of site design and development work, Performance Matters is now being transformed into a new journal with its own domain, at WebPerformanceMatters.com.

I built both sites myself using the Squarespace platform, and I'm now in the process of migrating all the old posts. Other than details of the progress of the migration, I won't be posting anything new here. But check out this post on the new site for all the details of the changes I'm implementing on the new platform.

See you there!
--Chris

Understanding Web Usability

2006-12-18T08:07:00.000Z

One of the great things about the Web has always been its democratic nature. Anyone can participate. But once you do, your contributions are wide open to public scrutiny. Good or bad, someone will evaluate your Web content.

People Recognize Poor Design
In the Web's early days, when people were adjusting to this new medium, most online critiques applied to a site's design. If you created a really ugly site, before long your handiwork would end being featured on a site dedicated to bad design, or even included on someone's all-time list of bad sites. Today, the collaborative features of the Web 2.0 environment such as blogs, forums, and widely used folksonomies practically guarantee that truly awful design will receive the recognition it deserves. Such criticism can be constructive; people can learn from good examples of what NOT to do.

Today, blogs and wikis extend their democratic and educational influences beyond site design to site content. This can be tough on the writer who wants to explore new ideas. But as Wikipedia, one of the best example of online collaboration, advises all contributors:

If you don't want your writing to be edited mercilessly or redistributed by others, do not submit it.

This is excellent advice, because the collaborative search for truth online is not subject to parliamentary or academic niceties. If someone advances a weak argument, people will be quick to point out its flaws. And an unpopular opinion can produce flaming responses.

Is Web Usability a Sham?
For example, last week Ryan Stewart, who has his own blog, and also writes a ZDNet blog about Rich Internet Applications (RIAs), wrote that Usability on the web is a sham, arguing that ...

While accessibility and standards are great for the Web, the concept of usability has been overblown. Usability as we define it is basically the rules for how the Web should behave while in the confines of the Web browser. But Web applications don't have to exist inside the browser and relegating them to these antiquated notions of usability is bad for progress.

To support this argument, he used the Back button as an example of a browser usability problem that RIA developers could eliminate altogether.

I think the central idea behind his argument was that the RIA technologies -- because they offer the developer more options -- can be applied to deliver usability improvements in Web-based applications. But I'm afraid he expressed it very poorly, first by over-generalizing in his criticisms of the concept of Web usability, and second by trying to use the Back button -- one of the most intuitive and widely-understood browser conventions -- as an example of poor usability.

Naturally, his post has been generating a small firestorm of responses ranging in tone from expletive-laden outrage to carefully-argued disagreement. In their different ways, both those examples (along with other responses) argue that Stewart's post is full of opinions and assertions that are not supported by any evidence, and that (to put it bluntly) Stewart doesn't really know what he's talking about.

Marshaling the Right Skills
Unfortunately, this is a common problem in the field of Web design. An effective online application must be available, responsive, easy to navigate, and deliver value to its user. This demands a wide array of skills rarely found in a single individual. As a result many sites are designed and built by people who are aware of only a small subset of the issues they should be considering. And all too often, someone in the site development process -- a business manager, someone in marketing, a Web designer, or a site developer -- makes key design decisions without understanding the consequences. And the challenges are even greater when developing a Rich Internet Application.

Simply getting more people involved isn't the solution. Some really bad Web site designs have been the result of design by committee. Even if you follow a good checklist, like this one by Vitaly Friedman, you will overlook some important aspect -- often site performance. The problem is the sheer breadth of knowledge required to do a good job. For evidence, read the Wikipedia entry on Web Design. It's an unbalanced, poorly organized, collection of information that offers little help with the process of creating an effective site.

The only answer is to get better, more knowledgeable, people involved. Start with a good overview of the process, like Usability for the Web: Designing Web Sites that Work by Tom Brinck, Darren Gergle, and Scott D. Wood. As they say in the book's introduction:

To ensure high usability on our own Web projects, we defined a development process that incorporates proven techniques from software and usability engineering, graphic design, project management, and other disciplines. The process had to be practical and lean. It had to allow us to work on multiple projects of varying sizes with fixed budgets. It had to help us keep track of the details that can kill usability and destroy profitability. This book is about that process.

Then find people who understand what the book is talking about to do the work -- and don't interfere!

Tags: Usability, Web design, Worst website, Ryan Stewart, Rich Internet Application, RIA, performance.

Web Design and Mouse Rage Syndrome

2006-12-16T02:17:00.001Z

Have you ever been frustrated at a Web site that downloads with the speed of an Alaskan glacier? Or become angry when a favorite site, or your Internet connection, is down. Have you experienced any of these symptoms:

Faster heart rate?
Increased sweating?
Furious clicking of the mouse?
Simultaneous clicking and cursing the screen?
Bashing the mouse?

Come on now -- admit it! Maybe some of them, just once or twice? Because if any of this sounds familiar, you're not alone.

Mouse Rage Syndrome
The consequences of poor Web site performance don't usually make news, unless there's a big outage on Black Friday or Cyber Monday. Then the story invariably focuses on the business lost by companies whose sites were overloaded or down. But what about the effects of poor performance on customers? A recent study by the Social Issues Research Centre (SIRC), an independent, not-for-profit organisation based in Oxford, UK., provides this perspective.

Researchers found that badly designed and hosted websites cause stress and anger, and coined the term "Mouse Rage Syndrome" (or MRS). They concluded that five key IT flaws in the way websites are designed and hosted may lead to harmful health effects.

Those five problems will come as no surprise to any regular user of the Web: slowly loading pages, confusing layouts, excessive pop-ups, unnecessary advertising, and site unavailability. But as I was saying in my previous post, it is not fair to blame IT for these problems, when most of them are the result of poor design choices by Web site designers and developers.

Damaging Customers' Health
The study combined data from a YouGov poll of 2,500 people with physiological tests on a separate sample of Internet users, who were asked to find information from a number of different websites. Tests measured physical and physiological reactions to website experiences, looking at brainwaves, heart-rate fluctuations, muscle tension and skin conductivity. According to the report:

When the test participants came to the 'problem' sites that we had deliberately chosen as comparisons for the 'Perfect Website' evaluation exercise [a prior study], responses changed quite dramatically in most, but not all, cases. While a few managed to stay calm and simply 'rise above' the problems presented by crazy graphics and slow-loading pages, others showed very distinct signs of stress and anxiety.

Some changes in muscle tension were quite dramatic...While this was happening, the participant's faces also tensed visibly, with the teeth clenched together and the muscles around the mouth becoming taught. These are physically uncomfortable situations that reduce concentration and increase feelings of anger.

These reactions, if not managed, can eventually lead to other, more serious, health problems.

Poor Designs Can Kill
According to the SIRC report, "users want Google-style speed, function and accuracy from all of the websites they visit, and they want it now. Unfortunately, many websites and their servers cannot deliver this". The result is consumers seeking alternative websites in a bid to avoid undue stress and Mouse Rage.

Jacques Greyling, managing director of Rackspace Managed Hosting, who commissioned the study, commented:

We believe that businesses that are selling online have a duty to their customers to ensure that the experience is as stress free as possible. The public has shown that it wants to buy online, ... (so) businesses need to provide simple and easy to navigate layouts, whilst focusing on speed and uptime.

If more studies confirm these findings, which will probably seem obvious to anyone who spends a lot of time online, maybe more Web designers will get the message that their poor designs are not just killing the business. They could be killing people -- really.

Performance Matters!

Tags: Social Issues Research Centre, SIRC, YouGov, mouse rage, Web design, performance, Web performance management, health.

Yahoo! on Web Page Performance

2006-12-12T08:00:00.000Z

A recent post by Tenni Theurer, who works in a performance team at Yahoo!, appeared in the Yahoo! User Interface Blog. The post begins with the claim that ...

... most of web page performance is affected by front-end engineering, that is, the user interface design and development.

Theurer introduces the Pareto Principle, commonly known as the 80/20 rule, which states that 80% of the consequences come from 20% of the causes. In the case of Web page download time, she argues that the backend systems which generate an HTML document -- apache, C++, databases, etc. -- should be regarded as the 80% of causes that account for only 20% of download time. The other 80% of download time is spent fetching the elements that make up the page, such as images, scripts, and stylesheets.

She presents two graphics to illustrate this point. A table summarizing home page measurements of 8 popular Web sites -- Yahoo!, Google, MySpace, MSN, eBay, Amazon, YouTube, and CNN -- shows that retrieving HTML accounts for between 5% and 38% of their download times, and just 12% on average. The other 88% is taken up downloading and rendering other page elements.

Waterfall Charts
To illustrate the reason why HTML accounts for such a small percentage of all page download time, Theurer uses the diagram shown here [click to enlarge]. I call this kind of data visualization a 'waterfall chart', because of its shape, and because it is similar to the way the waterfall model of software development has always been always depicted graphically.

Theurer says (in blog comment #22) she used the IBM Page Detailer tool to get the measurement data shown in this diagram. I have used this tool, and it produces a Chart Panel similar to the one above, which is a graphical representation of all of the component downloads that make up a single page.

I have not seen much documentation online, but if you take the time to register with IBM and download the software, its Help file is quite informative. It explains that the ...

... IBM Page Detailer attaches to the workstation TCP/IP socket stack and monitors all of the HTTP/HTTPS protocol based communications between your Web browser and the network. IBM Page Detailer then displays web activities in a series of color-coded bars.

It also has quite a long section on Results Analysis and other Considerations regarding Web download times.

This kind of analysis is not new. Since 1998, Keynote has offered a similar Web Content Diagnostics feature in its MyKeynote portal -- here's a datasheet describing the newest version of this service, recently released. For example, the figure on the right [click to enlarge] shows a Keynote diagnostic measurement of the Yahoo Search Marketing page.

Unlike the IBM Page Detailer, which always measures from the desktop where it is installed, Keynote's diagnostic measurements can be taken from anywhere on the company's worldwide network of 2500 measurement computers (or "agents"). This example comes from an agent located on the Verizon network in Houston.

The Lesson for Web Designers
Even without looking at the data in any detail, the message of these waterfall charts is evident from their general shape:

Web page download times rise with the number of bytes to be downloaded and with the number of separate content elements comprising a page.

This fact is not news to anyone whose job involves monitoring or managing site performance. Even for a Web page consisting only of a single HTML file, you cannot compute download time simply as page size divided by bandwidth. That's because of the way the TCP slow-start protocol works, sending one or two packets (typically 1.5Kb - 3KKb), then doubling, and doubling again, up to the receive window size of the client.

But most Web pages these days contain many imbedded graphic objects, CSS files, and script files, all of which extend download times. Most of these files are small, requiring just a few TCP packets to transmit. But every transmission requires a separate "Turn", adding another round-trip delay time (RTT) from the client (i.e. browser) to the server(s) and back. This behavior has been widely documented and is well-understood by performance specialists:

In a previous post here, I described the Web Site Response Time Model and linked to the CMG2000 paper (E-Commerce Response Time: A Reference Model) in which I introduced it.
In 2001, Peter Sevcik and John Bartlett of NetForecast published their paper on Understanding Web Performance, in which they first outlined a formula for computing Web page download times based on total bytes, the number of components, and on round-trip times between browser and server(s).
That formula was simplified and explained in a highly readable paper by Alberto Savoia -- Web Page Response Time 101 -- published in July 2001 in STQE, the magazine of software testing and quality engineering (now Better Software Magazine).

Since then, many articles and papers have reiterated this message. An entire industry segment -- Application Delivery Systems -- has emerged to help organizations offset delays in delivering their Web pages and satisfy customers' expectations of responsiveness. All the same, as is clear from the comments posted to the Yahoo! User Interface Blog, some Web designers and developers still need to learn this lesson.

Tags: Yahoo, Pareto principle, 80/20 Rule, performance, Web performance management, IBM Page Detailer,Keynote, page size, round trip time, RTT, TCP, slow start.

ITIL Crash Course

2006-10-27T05:24:00.000Z

Anyone involved with IT these days knows that ITIL is a hot topic, and one that seems to get hotter every month.

ITIL, the Information Technology Infrastructure Library, has evolved from work sponsored by the UK Government in the late 1980's. According to the official ITIL site, it is "the most widely accepted approach to IT service management in the world". For a lot more detail, see the wikipedia entry for ITIL, which describes it as "a framework of best practice approaches intended to facilitate the delivery of high quality information technology (IT) services", and goes on to explain that:

ITIL outlines an extensive set of management procedures that are intended to support businesses in achieving both quality and value for money in IT operations. These procedures are supplier independent and have been developed to provide guidance across the breadth of IT infrastructure, development, and operations.

If you subscribe to any technical magazine, newsletter or blog written for an IT audience, then you have probably seen more than one story on the benefits of adopting ITIL-based approaches to IT management. The latest publication to jump on the ITIL bandwagon is this week's InfoWorld (October 23, 2006, issue 43) -- also available online -- which contains an excellent 5-page article introducing ITIL.

The magazine cover touts the importance of ITIL today: To keep pace with business strategy, IT need a blueprint for delivering services. That's where ITIL, the modern-day bible of governance, comes in. The article itself is arranged like an interview in which the authors, Randy Steinberg and Michael Goodwin, answer ten questions. I won't list each question, because often you have to read the answers for the next question to make sense. But here's a summary.

Ten questions
Question 1 presents three examples to show how ...

ITIL raises customer satisfaction, reduces waste in the IT organization, and lowers operating costs.

Questions 2 and 3 introduce the processes that together make up the ITIL discipline of Service Support, and the related idea of a Configuration Management Database (CMDB) that provides the foundation for storing and retrieving all information about the IT infrastructure, and making timely management decisions.

Question 4 moves to the equally important discipline Service Delivery, and includes an informative diagram showing the central role of Service Level Management as the function that regulates the interface between Business and IT. [Incidentally, this highlights a perennial issue we face in high tech, where the ever-shifting meanings of common terms makes it hard for people to communicate clearly. In this instance, ITIL's definition of Service Level Management covers a narrower range of activities than my previous use of the same term here.]

Question 5 introduces the other ITIL books: Introduction to ITIL, Planning to Implement Service Management, ICT Infrastructure Management, Applications Management, The Business Perspective, Security Management, and Software Asset Management.

Question 6 asks, ITIL has been around for more than 15 years. Why is it now taking hold in the United States?, and answers:

The United States has always lagged behind Europe in the discipline of IT infrastructure management. Go to a bookstore, and you'll find precious few books on the subject. Go to the universities, and you'll visit a lot before finding one that teaches it. One reason that this is changing is the increasing impact of European- and Asian-owned companies operating in the United States, all saying, Hey, get on board! You need to be ITIL-compliant. That's how we run our shops over here.

It also refers to something I have written about here. Whereas IT customers of the past were "internal staffers, managers, and auditors" ...

these days increasing numbers of IT customers ... are external -- actual customers of the business itself, interacting via public Web sites. If systems fail, potential buyers are likely to take their business elsewhere; potential damage to corporate reputation is high.

Question 7 deals with the needs of large and small IT shops.

Questions 8 and 9 address getting started and getting outside help with your ITIL implementation.

Question 10 talks about ITIL's relationship to ISO 20000. I will discuss the relationship between ITIL, COBIT, and some ISO standards in a future post.

Summing up
The article concludes that ...

Companies that have initiated ITIL efforts are already seeing higher customer-satisfaction levels and reduced costs and labor. Although not a panacea for all IT challenges, ITIL is a fundamental conceptual change for how IT will be doing business in the 21st century. Its time has come.

From all the evidence I see, this is an accurate summary. If ITIL's time has not yet come for your organization, maybe you should expect it to arrive soon. In future posts, I'll be writing a lot more about how ITIL applies to Web Performance Management. But if you just finding out about ITIL, I recommend this InfoWorld article as a good starting point.

Tags: ITIL, InfoWorld, ISO 20000, performance, Web performance management, service level management, SLM.

Web Performance Engineering [4]

2006-08-28T04:01:00.000Z

Continuing my series of posts on Web performance guidelines, today I'm reviewing one chapter of a new book, Deliver First Class Web Sites: 101 Essential Checklists, by Shirley Kaiser of SKDesigns, published by Sitepoint in July 2006.

Sitepoint is known among Web developers for its practical publications, and Kaiser deserves credit for including a full chapter on site performance alongside all the other useful advice in this book.

As its title states, this book does not cover topics in great depth. Each checklist item is stated, then discussed briefly. But Kaiser does include enough detail to justify each recommendation, and often includes examples of how to do it. Chapter 11 (Web Site Optimization) is 19 pages long, presenting 41 recommendations arranged into six checklists:

Creating Clean, Lean Markup
Minimizing URLs
Optimizing CSS
Optimizing JavaScript
Supporting Speedy Server Responses
Optimizing Images, Multimedia, and Alternative Formats

However, once I started reviewing Kaiser's checklists, I soon noticed a remarkable similarity between her choice of topics and those covered in Andrew King's book, Speed Up Your Site -- the subtitle of which also happens to be Web Site Optimization.

A bit more scrutiny, summarized in this spreadsheet, revealed similarities in the naming, wording, and organization of the checklist items too. Although Kaiser cites Speed Up your Site in her first paragraph, she does not actually recommend it explicitly, or mention any collaboration with Andrew King. But she seems to have relied heavily on that one source as the main inspiration for her Web Site Optimization chapter.

To give Kaiser her due, rather than simply parroting all of King's recommendations, she has ignored the more extreme ones. All the same, her material still shares most of King's limitations and omissions, many of which I listed earlier when I reviewed his book, and which I plan to cover in future posts in this series.

In conclusion, if you'd like a handy summary of the material in King (except for his excellent discussion of the Psychology of Performance), Kaiser's book has it, plus about 300 more pages of useful checklists on other topics. I recommend it, because I don't think you can go far wrong with any book from Sitepoint. If you want to read more about the same topics, you can find Speed Up your Site on sale on Amazon these days under half price. But don't assume that either book will give you a well-rounded picture of Web Site Optimization issues and techniques.

For more ideas about that, continue reading Performance Matters, and I'll do my best to fill in the holes.

Tags: Shirley Kaiser, SKDesigns, Sitepoint, performance, Web site optimization, Checklist, page design, site design, service level, SLM, optimization, Web optimization, tuning, Web tuning.

Are Online Retailers Ready for Business?

2006-08-25T05:31:00.000Z

Every year, more and more shoppers turn to the Web for their holiday shopping, with total sales in 2006 projected to be in the multi-billion dollar range. But will online retailers be up to the task?

My team at Keynote recently studied 25 top online retailers in three categories: Books and Music, Electronics and Apparel. The study involves measuring a typical customer's navigation path through each site -- from Home Page, to Search, to Product Details, and finally Checkout.

Using computers ("agents") that emulate the behavior of a customer using an IE browser, we measured the exact same sequence of steps on each site from 10 locations throughout the US, over various connection speeds, every hour for a month (mid-May through mid-June, 2006). This produces a large data sample (over 6,000 data points per site), which we then subject to a lot of statistical analysis. By the time we publish the report, we believe we have a reliable picture of a site's health and readiness.

This is the second year we've conducted this particular study, and the 2006 results were surprising. While the top-ranked sites continue to provide excellent service -- almost perfect availability, excellent download speeds, and very little inconsistency -- the lowest-ranked sites had some serious failings. Without doubt, the performance problems we saw were bad enough to dissatisfy customers and impact the bottom line. They fall into three general areas:

Outages:
We consider a transaction (the sequence of steps) to be unavailable if any part of the purchase path fails so that the customer is unable to complete their transaction. And when 30% or more of our measurements during an hour report a site as unavailable, we count it as an outage.

Only 4 of the 25 measured sites registered no outages during the month, while some of the least reliable sites had more than 15 hours of downtime. These hours are during the peak period each day (8 a.m. to midnight EDT). Apparel sites were especially outage-prone, averaging over 4 outages each.

We all know how frustrating it can be to try to buy something online only to have the process fail midway through, or to not even be able to get to the site's Home Page. I'm not sure which is the more annoying. And if these major retail sites can't stay up under relatively light summertime loads, how will they respond when their traffic increases dramatically during the holiday season? In the increasingly competitive retail marketplace, being down for even a single hour at that time of year can have a significant financial and brand impact.
Load Handling:
One area of our analysis involves a site's ability to keep up with its current load, without any performance degradation. On a site that is built with sufficient capacity, performance does not change noticeably as traffic fluctuates during the day. At this time of year, most retail sites should be idling, because traffic is light. Indeed, we saw that the best sites, Barnes and Noble and Gap, had virtually no slowdowns each day, indicating that they can handle their present load comfortably.

Of course, it takes a controlled load-testing project to discover what happens when traffic volumes are doubled, tripled, quadrupled, and so on. But judging from our measurements, these sites do appear well prepared to handle increases in their load. In contrast, several sites already display significant load-handling issues now, slowing down as much as 100% each day. For example, a page that normally appears in 3 seconds would take 6 seconds under load. Not only does this kind of sluggish behavior annoy current customers, it suggests that these sites are highly likely to crumble under the increased load, once holiday shoppers begin to fill their online storefront -- unless something is done before then to increase capacity.
Dial-Up Performance:
While many of us can't remember the last time we used dial-up, a large percentage of users still use slower connections. These might not be dial-up, but a poor wireless connection, a slow hotel connection, or a satellite connection may not be much faster. So site designers shouldn't forget about their bandwidth-challenged customers.

Yet that's exactly what seems to be happening. Many retail sites seem set on alienating the dial-up user. The sites we tested averaged 30-40 seconds per page, which really adds up fast in a 10-page shopping transaction. And in some cases we even clocked Home Pages taking over 100 seconds to download. A few sites got it right, Dell being a good example. They have the fastest Home Page in the study for dial-up users because they serve a slightly trimmer page for this audience.

In short, the best sites are ready, and their performance sets the standard for the retail industry. But even among the leading retailers we studied, many could do a lot to improve the quality of the online shopping experiences they offer.

More importantly, if one of these sites is struggling now, what will happen in December, when I desperately need to buy my last-minute gifts? I'm not optimistic they will be ready to take my business. And I absolutely hate fighting the crowds at the local shopping mall, so -- along with thousands of others like me -- I'll be buying from their competitor's site.

Update #1: To hear a longer (12.5 min) discussion of these results and related issues in the retail industry, listen to a StorefrontBacktalk Week In Review Audiocast discussion I took part in. Click on the link for the section on whether the major E-Commerce sites are ready for the holiday rush or the holiday crash.

Update #2: I was also interviewed about the study today (8/25) on CNBC's Closing Bell program. As you might expect, that interview was a lot shorter than the StorefrontBacktalk Audiocast. You can view a replay online; the username is keynote and password is keynote0895.

Tags: ecommerce, online retail, holiday shopping, performance, Web performance, site outages, load handling, load testing, service level, dial-up.

Web Performance Engineering [3]

2006-08-22T07:01:00.000Z

Continuing my series on Web performance guidelines, today I am reviewing another book -- Speed Up Your Site, by Andrew B. King, published by New Riders in 2003.

A while back, when I was reviewing Web Usability Books, I promised to cover Speed Up Your Site, but never got around to doing so -- for reasons I will explain. A full table of contents listing all 19 chapters is available online; in summary, the book has six parts:

Part I - The Psychology of Performance (38 pages)
Part II - Optimizing Markup: HTML and XHTML (99 pages)
Part III - DHTML Optimization: CSS and JavaScript (111 pages)
Part IV - Graphics and Multimedia Optimization (85 pages)
Part V - Search Engine Optimization (39 pages)
Part VI - Advanced Optimization Techniques (79 pages)

Part of my difficulty in reviewing this book was that I have mixed feelings about it. Naturally, I am always pleased to see an entire book devoted to performance issues. Also, Part I is particularly good. Containing 77 references, it is a very well-researched survey of an important subject rarely covered in books about Usability. On the other hand, I am not nearly as impressed with the rest of its advice and guidelines, for two reasons: their correctness, and their completeness.

First correctness. Regarding the actual content, my issue is not that Speed Up Your Site contains factual errors. As far as I can tell, its content is accurate. But some of its recommendations -- although they may indeed improve performance -- are so unnatural that they are not really the correct way to tackle the problem. This concern is summarized by Alexander Bunkenburg in this review on Amazon.com:

This book concentrates almost exclusively on sending fewer bytes from the server to the browser. It gives a large collection of tricks how to write shorter html, xhtml, css, and JavaScript. Some of these tricks are useful. Others however go against standards, and some seriously go against maintainability. I'd be reluctant to give this book to my team. One may be tempted into shaving off bytes, spending a big effort and yet producing unmaintainable code.

The problems that can result from deliberately violating standards are highlighted by David Rose in another Amazon review:

In today's world, where "standards based" coding is becoming more prevalent and adherence to the W3C standards for HTML coding is being recommended, this book just grated on me. While there is a great deal of great information, there are also a large number of "gotchas" to watch out for as well.

The book proposes to use HTML tags without their corresponding closing tags, not to use required elements whenever possible, avoid using quotes in HTML tags, and many other ways of creating "non-valid" code. This will "optimize" your code a bit more by reducing the characters in it, but it will also create problems for you in the future.

In summary, while the book does give a lot of good information, it often steers you away from standard code. If you are unsure what is considered "standard" and required for creating valid XHTML/CSS, you are best served skipping this book as it will teach you to create invalid code.

Bunkenburg's review also touches on my second area of concern -- completeness. Any book about speeding up Web site performance should (in my view) at least mention all the important topics, to let the reader know what options exist. Some important topics that receive little or no coverage in Speed Up Your Site are:

Page Types: All pages are not created equal, and users' tolerance for delays changes depending on where they are in their interaction with a site. So a single page design approach cannot be applied to all pages.
Maximize content reuse: A common mistake as sites grow is to use many different names (URLs) for the same thing, reducing the efficiency of browser caching.
Ratio of HTML base to content: There is a significant difference between the way browsers handle the base (or index) portion of the page, and referenced content elements, which can affect page download time.
Image resizing: For efficient page rendering, it helps to specify HTML HEIGHT and WIDTH tags for all embedded images. And for download speed, avoid resizing images in the browser.
Performance of SSL: All online business sites use encryption, and pages that use SSL encryption incur significant overheads. This is an issue that demands a separate section in any book about site performance.
Content Delivery Networks (CDNs): Akamai went public in 1999, and was widely used by 2002. Mirror Image, Cable and Wireless (formerly Digital Island), and Speedera also offered CDN services in 2002. How can any book about speeding up your site not even mention this technology?

In fairness to King, any discussion of the book's coverage should point out the following disclaimer, which appears in its Introduction:

Although the primary emphasis is on optimizing client-side technologies, this book also covers server-side techniques and compression to squeeze the maximum performance out of your site. These are all techniques that most designers and authors can control. Instead of focusing on esoteric server-side tuning limited to system administrators, this book focuses on optimizing the content that you deliver. For a server-oriented look at performance, see Web Performance Tuning, by Patrick Killelea.

But most of the omissions I listed are not related to server-side tuning, and of the two that are (SSL and CDNs), only SSL is discussed by Patrick Killelea, and his only recommendation is to use an SSL accelerator card. And in 2002, CDN technology could hardly be considered "esoteric", especially in a book which, to quote its author, is "not for beginners".

There is a lot more to be said on most the above topics, and in future posts I will expand upon them.

Tags: Andrew King, Speed up your site, performance, Web site optimization, page design, site design, service level, SLM, optimization, Web optimization, tuning, Web tuning.

Web Performance Engineering [2]

2006-08-18T07:01:00.000Z

Today I'm going to look at another list of Top Ten Web Performance Tuning Tips, following up on my promise to review Web site and application performance advice.

Today's list of tuning tips was created by Patrick Killelea, the author of Web Performance Tuning, first published by O'Reilly in 1998, then revised in 2002. When the second edition came out, Patrick also updated his 1998 top ten list, presumably to reflect changes in the rapidly maturing Internet and Web environment. But O'Reilly still publishes the 1998 list alongside the 2002 list without any further explanation, even though just four recommendations appear on both lists!

I see this as evidence that publishers are a lot more interested in selling a book than they are in the usefulness of its content. So let's blame O'Reilly and give Patrick the benefit of the doubt here, and focus on his latest list only. Abbreviating his recommendations, they are:

1. Check for compliance with standards
2. Minimize use of JavaScript and style sheets
3. Turn off the Web server's reverse DNS lookups
4. Try out a free analysis tool (to find bottlenecks)
5. Use simple servlets or CGI
6. Get more memory
7. Index your database tables well
8. Make fewer database queries
9. Look for packet loss and retransmission
10. Monitor your Web site's performance

When someone publishes a top ten list, I expect it to include the ten most important and useful recommendations -- especially when its author has written the most comprehensive book available on the subject. In this case, even allowing for the maturing of the Web since 2002, I have no idea how Patrick could have come up with this list. I have three problems with it -- what's in it, what's not in it, and its order. Today I will tackle mainly the first area; here are some brief thoughts about each of his recommendations:

Check for standards compliance by using Weblint or other HTML checking tools.
Content that conforms to the HTML 4.0 standard will load faster and work in every browser because the browser then knows what to expect. Note that Microsoft-based tools create content that does not even use the standard ASCII character set, but instead uses many proprietary Microsoft characters that will display in Netscape as question marks and can slow down rendering.

performance

Watch out for Composition Tools with a Bias

Minimize the use of JavaScript and style sheets.
JavaScript is a major source of incompatibility, browser hangs, and pop-up advertising. Style sheets require separate downloads before the page can be displayed. There are some nice features to both JavaScript and style sheets, but at a big cost. Life is better without them.

Turn off reverse DNS lookups in the Web server.
If left on, reverse DNS will log a client's machine name rather than IP address, but at a large performance cost. It is better left off. You can always run log analysis tools which look up the names later.

in 1997

Prior to Apache 1.3, HostnameLookups defaulted to On. This adds latency to every request because it requires a DNS lookup to finish before the request is completed. In Apache 1.3, this setting defaults to Off.

Try out a free analysis tool.
I've provided a free analysis tool at my Web site that can tell you whether or not your bottleneck is in DNS, or because of connection time or content size, or is on the server side. Work on improving the slowest part first.

a page about his book

Use simple servlets or CGI.
Use simple servlets, CGI, or your Web server's API rather than any distributed object schemes like CORBA or EJB. Distributed object schemes are intended to improve a programmer's code-writing productivity, but they do so at an unacceptable cost in performance for end-users.

Get more memory.
Your Web server, middleware, and database all will probably do better with more memory, if they still use their hard disks frequently. Hard disks are literally about a million times slower than memory, so you should buy more memory until the disks are phased out.

Index your database tables well.
Spectacular improvements are possible if you are inadvertently doing full-table scans on every hit of a particular URL. Indexes allow you to go directly to the data you need.

spectacular improvements

Make fewer database queries.
If you can cache content in your middleware or servlets, do it. Making connections to a database and using those database connections is typically a bottleneck for performance.

doing less work

any

Look for packet loss and retransmission.
There are many network snooping and monitoring tools to help you do this. Intermittent slowness is often due to packets being lost or corrupted. This is because a time-out period needs to pass before the packet is retransmitted.

Set up monitoring and automated graphing of your Web site's performance.
This information is free online in Chapter 4 of the second edition of Web Performance Tuning.

you can't manage what you don't measure

researching it

So, to sum up my audit of Patrick's list of ten guidelines, I vote to reject two altogether (#1 and #2), downgrade one (#3) to a priority well outside my top ten, accept four (#4, #5, #6, and #7), restate two (#8 and #10) as general principles that don't belong on this list, and reserve judgment on one (#9).

That opens up 5 or 6 slots for the things that Patrick missed -- but what should they be? I will tackle that subject in a follow-up post.

Tags: Killelea, Patrick Killelea, O'Reilly, performance, Web performance, page design, site design, service level, SLM, optimization, Web optimization, tuning, Web tuning.

Baseball and the Price of Gas

2006-08-17T05:35:00.000Z

This is only tangentially related to the usual subjects I cover in this blog, but it certainly relates to the way I approach research and blogging. I am always doing research online, and during summer evenings and weekends that activity is often accompanied by the day's radio broadcast of the Oakland A's baseball game -- the best baseball team here in the San Francisco Bay Area, by any objective standard.

Tonight was no different, and A's broadcaster Robert Buan caught my attention when he opened the post-game show. He pointed out that in winning tonight, the A's have secured their biggest lead in their division since September 30, 1992. As a fan of the team, this is interesting; to everyone else it's probably instantly forgettable. But more intriguing was what he actually said -- in the very first sentence of the program. He opened his show with this statement:

According to a reliable authority, wikipedia.com, the last time the A's had a lead of six and a half games in the American League West, gas was selling for $1.38.

I couldn't help reflecting on how much the Web is changing the way everyone approaches information and research!

Reporting Web Application Responsiveness

2006-08-15T17:45:00.000Z

In a previous post, I discussed some complications of measuring Rich Internet Applications (RIAs). In particular, I concluded that …

… to report useful measurements of the user experience of response times, instead of relying on the definition of physical Web pages to drive the subdivision of application response times, we must break the application into what we might call logical pages. To do this, a measurement tool must recognize meaningful application milestones or markers that signal logical boundaries of interest for reporting, and thus subdivide the application so that we can identify and report response times by logical page.

Today I am going to look inside the logical page, and consider what happens when the application responds to a user action. I have previously written about the stages of this process here, and in this paper.

Two Standard Service Level Metrics
Because Web pages are constructed using many separately downloaded components, low level monitoring tools can collect a ton of data about the communications activity triggered by a user interface action. And for diagnostic and tuning purposes, it is always useful to measure each separate component of the download process. But for most service level monitoring, this low level data can usually be summarized in just two important metrics: initial response time and page download time.

From the perspective of a typical user, these metrics represent two distinct events in the page download experience. After 'clicking' on a page, the first measures the apparent pause until the site begins to respond, while the second measures the time for that response to download.

The elapsed time between these two events can be large enough to affect the way pages are designed. Because pages containing many images take a while to download over slower connections, such pages are normally laid out so that the most popular links are displayed early in the page download process, allowing a user to navigate to another part of the site without having to wait for the all the page content to complete.

But while initial response time and page download time provide a good indication of the user’s experience of a traditional Web application, they may be less useful for some RIAs. In these applications, the time to complete a page download might include additional asynchronous activity that does not affect the user’s experience of the current page, such as fetching additional content (like executable script files, application data, images, or even streams) for future use within the application.

An Additional Metric for RIAs
This suggests that, while the two existing metrics are still useful, a standard scheme for measuring and reporting RIA responsiveness probably needs to also include a metric that represents the time from user interface action until an intermediate event, one that we might call page display complete. This metric would reflect the time it takes to download everything needed to complete the display of the current page.

Even that definition is ambiguous, because it may not be obvious when some pages are 'completely displayed'. I see two issues here. First, what about parts of the page outside the display window, that won’t be visible until the user scrolls? Let’s deal with that by assuming the user has a screen of unlimited size, so that the entire page could be visible without scrolling.

Second, what if a page includes a streamed video, or a series of images that are displayed in sequence, each replacing the previous one? On reflection, I conclude that the 'page display complete' event should occur when the complete display is first visible -- in my examples, when the video begins playing, or the first image is displayed. That's because measuring to the last view of the display would make this metric a lot less useful as a measure of a user's experience of responsiveness. And nowhere between those two extremes offers any possibility of generality across applications.

Summarizing What’s Needed
So now I have defined three distinct metrics that seem to have fairly universal applicability when measuring the responsiveness of RIAs:

Initial response time
The elapsed time from a user interface action (a mouse click, or any other user action that triggers a download process) until the browser places the page in a state that permits a user to perform another action (technically, the user is unblocked). We could also say that this is the time until the new page first becomes usable.

Page display time
The elapsed time from a user interface action until the complete resulting page is first displayed, or could be first displayed if the screen were large enough.

Page download time
The elapsed time from a user interface action until the complete resulting page and all associated content elements have been downloaded. This time includes all secondary or background downloads triggered by the user's action, whether or not they are required for the current page display.

These definitions provide only a rough statement of requirements for RIA measurement -- none is technically precise. In practice, any attempt to implement any one of these requirements could present problems for some applications, browsers, or measurement tools, and I will write about some of those complications in a future post. But I think it is useful to identify these generally applicable goals for measuring and reporting on the responsiveness of Rich Internet Applications.

Tags: RIA, Rich Internet Application, measurement, service level, SLM, page download, Web application, Webapp.

Web Performance Engineering [1]

2006-08-15T00:12:00.000Z

This post is the first in a new series on how to build Web pages, sites and applications that perform well -- by design. I will be combining my own observations with online research and recommendations on best practices contributed by my colleague Ben Rushlo. Ben makes his living measuring Web site performance and giving companies advice on how to improve the performance of their sites and applications.

Despite the crucial contribution of performance to online application effectiveness, good advice on the subject is surprisingly scarce on the Web, and even scarcer in books about Web design. Although you do sometimes find small sections (rarely chapters) of such books devoted to performance, their content is usually weak and almost never a systematic treatment of the issues. I won't bore you with proof, but I do mention one example below.

On the other hand, the exceptions are certainly worth noting. Recently I found an excellent discussion of 10 Realistic Steps to a Faster Web Site, posted in February 2006 by Alexander Kirk, a Web application programmer in Vienna, Austria. I mention his profession because in my experience, a lot of advice about performance written by programmers completely ignores the central rule of all performance optimization -- to speed anything up, remove the biggest bottleneck.

Applying just this one rule repeatedly (think about it) will always produce the best results. And in the world of distributed systems, the biggest bottlenecks will almost always be related to the way an application uses a network, not to the way a computer (client or server) processes some program code. In other words, the application's logic is usually far more important than the way a programmer coded it.

Kirk obviously understands this. After complaining about an example of misguided performance advice (read the comments), he presents his view of a systematic approach to Web application performance analysis. I will not attempt to summarize it beyond simply listing the outline, which is:

1. Determine the bottleneck
1.1. File Size
1.2. Latency
2. Reducing the file size
3. Check what’s causing a high latency
3.1. Is it the network latency?
3.2. Does it take too long to generate the page?
3.3. Is it the rendering performance?
4. Determine the lagging component(s)
5. Enable a Compiler Cache
6. Look at the DB Queries
7. Send the correct Modification Data
8. Consider Component Caching (advanced)
9. Reducing the Server Load
9.1. Use a Reverse Proxy (needs access to the server)
9.2. Take a lightweight HTTP Server (needs access to the server)
10. Server Scaling (extreme technique)

Kirk's discussion does omit a few topics (subjects for future posts in this series) that I think are important, but this is an excellent starting point. While his wording implies a reactive approach to his subject matter (writing about tuning or improving the performance an existing site), most of his guidelines relate to best practices in site design and engineering. So there is no reason why they should not be implemented proactively, without waiting for problems to surface.

Tags: Web performance, page design, site design, service level, SLM, optimization, Web optimization.

Rich Internet Applications: an update

2006-07-23T06:23:00.000Z

Blogging has taken a backseat to other activities for a couple of months, so I have some Performance Matters to report.

Following publication of the seventh in my earlier series of posts about managing Rich Internet Applications (RIAs), I reworked most of that material into a single paper. After some delays to reformat the content and redraw the diagrams, it is now available an an Industry Brief, which you can download from the Keynote resource library. The title is Rich Internet Applications: Design, Measurement, and Management Challenges.

Although I referenced Wikipedia when introducing the subject of RIAs, and retained those references in the paper, I did not feel that the current contents of Wikipedia did a very good job of explaining RIAs. In my second post I described how I had met Aleksandar Šušnjar online after reading his contributions to the discussion of RIAs in Wikipedia. I wrote that Aleks had:

... shared insights gained from his experience building a product first released in 2002 ... In delivering desktop capabilities via the Web, it clearly predated much of today's thinking about the purpose of RIAs and how to build them.

Naturally, Aleks also discovered some potential pitfalls an RIA designer might run into, which I will be mentioning in later posts. Regrettably, many of his insightful contributions to Wikipedia (which can still be found in its history pages) were subsequently removed by other contributors who lacked either his intimate knowledge of the technology or his historical perspective. Rather than waging Wikipedia edit wars, which for a hot topic can continue interminably, he has simply posted his own page about RIA and AJAX.

Despite Aleks' experiences, having made the effort to understand and summarize the SLM challenges posed by RIAs as coherently as I could, I felt it was time to try my hand at editing Wikipedia. So I have tackled the Wikipedia RIA page; I suspect that this entry may not be quite as "hot" as the Ajax entry, where Aleks was posting. So today (at least) you can find my handiwork throughout the section that compares RIAs with traditional Web applications, including (not surprisingly) the entire section on management complications.

Tomorrow may be another story. After all, the Wikipedia editor carries the clear warning: If you don't want your writing to be edited mercilessly or redistributed by others, do not submit it. So please feel free to improve on my efforts -- I know what to expect!

Tags: RIA, Rich Internet Application, management, wikipedia, service level, SLM, Web application, Webapp.

Insights from Interop 2006

2006-05-02T19:32:00.000Z

In my last post I promised that if I get any new insights worth sharing this week while I'm at Interop 2006, I'll write about them. Well, here's the first installment, which consistes of just three items. I'm supposed to be at the Keynote booth on the show floor soon, so this will be short, because it's quite a hike to get there.

I'm staying at the Luxor, a hotel cleverly designed to resemble an Egyptian pyramid. Egyptian motifs abound. To reach the Conference Center in the Mandalay Bay hotel, which is next door, you can take a cab, a shuttle bus, or ride on a tram. But I usually walk. I hike about half a mile through a network of corridors cleverly disguised as places to spend money -- two casinos, a shopping mall, theaters and entertainment centers, restaurants, etc. I know I need to exercise more, so this a token effort.

Back to the insights. Yesterday afternoon I attended the Application Performance Day. Peter Sevcik of NetForecast presented a very thorough analysis of Performance Improvement Solutions, covering QoS, Compression, CDN's and other Acceleration technologies. He also provided a sheet of references, some of which I will publish when I have more time. Two insightful comments I wrote down were:

Fast delivery of sites results in slow delivery of pages. We all know that when the development process does not include time to think about performance, the results are not optimal. I thought Peter's simple rule captured this issue in a memorable way.
Most developers build applications in a LAN-based environment, and don't insert a WAN simulator to test their performance. Another well-known problem nicely summarized. If you don't actually test your application in the production environment, or something that resembles it, you will surely run into performance problems in the real world.

Which brings me to my third item. Last night I dreamt I was one of a small group of people lugging a grand piano from the Luxor to the conference center. But instead of taking the inside route, we were outside on the street. And every time we came to a gap in the sidewalk, several of us had to lift the piano over the kerbs. This was becoming hard work, and so at one point Steve Jobs, who owned the piano, wanted to leave it where it was. But John Chambers (CEO of Cisco Systems), who seemed to be supervising the move, said "No Steve -- we can't just leave it here, we have to deliver it."

Now I'm not sure what lessons (if any) I should draw from this dream. It certainly seems to be a metaphor for many of the discussions going on here about how to deliver large payloads to remote destinations via less than ideal routes! Maybe John Chambers actually offered some more concrete suggestions during his Interop keynote speech this morning; I was still in my hotel room moving the piano at that time.

Now I'm going to mingle with the crowds and find out if Interop can make me smarter today.

Alistair Croll on Ajax

2006-05-02T09:24:00.000Z

Because of my particular interest in software performance optimization and SLM, I often search the Web to see what people are writing about performance. And even though there are always new things to find, it seems to be getting harder to locate them.

One reason is that until recently, anyone writing about the performance of computer systems or applications was discussing SLM topics like speed, availability, optimization, capacity planning, or load testing. But in the last few years, as the Web has made information systems and business systems synonymous, the term performance has gradually taken on a much broader meaning. Now to locate the interesting links, I have to include technical keywords that are designed to filter out unwanted information about business success.

So when I do find something worth reading, I assume it's a good idea to make a note of it, to save someone else the trouble. I have already included the most useful references in my previous posts about the performance of Rich Internet Applications (RIAs), but yesterday I found this excellent blog post by Alistair Croll of Coradiant on the impact of AJAX on web operations. From there, I also located his related presentation on Ajax and Networks to Interop in December 2005.

Finding this material was a complete coincidence, in that I happen to be attending Interop 2006 in Las Vegas this week, yet I found that link independently. But it was a good very thing to know about, because on Wednesday I am speaking at the WebOps Summit (scroll down to Wednesday for details), which is chaired by none other than ... Alistair Croll. And I have actually devoted the last third of my talk on Best Practices to the emerging RIA measurement issues I discussed here. Even though I have been researching and writing about the performance implications of Ajax and RIA's in general, and scheduling this Interop presentation with Alistair, I had still missed this connection.

This illustrates just how hard it is these days to stay on top of any topic being discussed online. So I hope this post will help you in that regard. And if I make any new connections worth sharing this week, I'll be writing about them when I get a chance. The cover of the conference guides say Interop Makes You Smart, so I'm hoping that works on me. Those marketing slogans are always true, aren't they?

Software Engineering Matters

2006-04-16T22:13:00.000Z

With the obvious exception of a few bloggers like Markos Moulitsas Zúniga, whose blog Daily Kos receives over a million visits per day, I think most bloggers are happy just to know that someone cares enough to read what they write. In my own case, having spent my career working on software performance, I learned long ago that only a small fraction of the population is actually interested in Performance Matters. So I don't expect a lot of feedback.

All the same it's nice to hear from a reader occasionally, and yesterday was one of those days -- I received a comment from Damith C. Rajapakse.

Curious about why a post on Waterfall Methods would be the one to generate a response, I did some digging. For readers who would like to hear about interesting stuff for the serious software engineer, Damith's blog -- Another Day in Mythical Man Month -- is worth a look. In particular, his recent post on maintaining future web applications (subtitled 'it's time to brace ourselves') is very relevant to my current focus on Rich Internet Applications. Damith notes that while ...

rapid development of Web apps is receiving the lion’s share of attention, maintainability of Web apps (is) hardly receiving a thought.

He warns about three trends that are likely to make Web applications hard to maintain:

The rush towards visual programming
Overuse of frameworks
Overdoing that 'AJAX ' thing

While it's good just to know that someone is reading what I write, it's even better when they have something useful to contribute to the discussion. And in this case I agree completely with Damith's perspective on the issues. Ten years ago, on the very first page of the introduction to my book about client/server performance, I wrote:

In today's world of visual programming, rapid development, and out-of-the-box solutions ... performance is one area of software design that is frequently misunderstood, forgotten, ignored, or postponed, often with disastrous results.

The same can be said of maintainability -- which, like performance, is always less interesting to software developers than creating new application function. That is why your software development process must ensure that characteristics like performance and maintainability are designed and engineered into applications from the beginning. In other words, it is important to realize that good Software Engineering involves a lot more than just programming and testing your code. According to Wikipedia, it is ...

... the practice of creating and maintaining software applications by applying technologies and practices from engineering, computer science, project management, application domains and other fields... Like traditional engineering disciplines, it deals with issues of cost and reliability ...

As I have noted previously, the complexity of Rich Internet Applications makes it essential to actually practice rigorous software engineering, and not just crank out code.

Waterfall Methods: Past and Ever-Present

2006-04-16T05:17:00.000Z

In my previous post, I commented that adopting a waterfall development methodology was not going to work well when developing Rich Internet Applications, because more agile methods were needed. While writing the post, I did a bit of research into the literature about Waterfall methods, a concept I clearly recall encountering for the first time in 1978 or '79 soon after I joined IBM's DB2 development team in the Santa Teresa Lab.

My own introduction to the Waterfall method was in a talk about its limitations. So it was interesting to read Tarmo’s psycho and techno blog explaining that the very first description of the model by Winston W. Royce in 1970 was also as a counterexample. This point of view is amplified by Conrad Weisert, who claims that in fact there's no such thing as a Waterfall Model!

But despite the fact that (maybe) it should not exist, the Waterfall model has taken root. This is evident from the fact that it merits a place in methodology comparisons and analyses. What's more, despite the many rumors of its demise, especially in the face of today's need for more agile methods, researchers have discovered that Waterfall methods refuse to die:

... in a survey of almost 200 practitioners, accounting for several thousands of projects over the past five years, the dominant process model reported was the Waterfall, with more than a third claiming its use.

Indeed, perhaps the ultimate tribute to the sheer persistence of the Waterfall approach in everyday development practice is the fact that the agile development community recently promoted the Waterfall 2006 Conference. Even if you couldn't attend, I recommend reviewing the agenda and the session descriptions carefully, just to see what you missed. I don't know if or when more documentation will be published, but I'm sure it will be worth waiting for.

Managing Rich Internet Applications [7]

2006-04-10T07:11:00.000Z

This is the seventh post in a series devoted to the challenges of Service Level Management (SLM) for Rich Internet Applications (RIAs). In these applications, some processing is transferred to the Web client while some remains on the application server. Previous posts introduced the subject and the SLM topics I plan to address, reviewed RIA technologies, introduced The RIA Behavior Model, and introduced and discussed RIA measurement challenges.

RIA Usability and the Site Development Process

Having discussed the measurement challenges posed by RIAs, I intend next to consider the demands that RIA development may make of SLM processes that have been designed and fine-tuned to manage traditional Web applications. I covered some of this ground in a Webinar on How to Overcome the Challenges of Rich Internet Applications. So in this post I will lay out the general framework of that material, using the illustrative figures I created for the Webinar.

In previous posts I have introduced and discussed a simple four-part framework for thinking about Web Site Usability. I divided usability into four dimensions, each of which is essential to the ultimate success of any site -- Availability, Responsiveness, Clarity, and Utility.

The graphic presents the four essential qualities in the order of their significance, at least for all first-time visitors to a site.

Interestingly, companies developing Web sites or eBusiness applications tend to approach these four aspects in exactly the reverse order, as shown in the next figure.

A business goal leads to a design process, which in turn results in an application being built and then rolled out into production. And at each point along the way, people do their best to make the outcome a success. But is that enough?

Well, if you take this approach you may get lucky. But to be safe you really have to take a more systematic approach to Service Level Management, like ITSO.

[On this subject, it is hard for me to be brief, because 10 years ago I actually wrote a 750-page book about it. But here goes!] I have always believed that you cannot separate issues of software performance from the software development life cycle (SDLC). The best way to ensure acceptable performance is to follow the discipline of Software Perfrmance Engineering -- to build in the desired level of performance, just as a mechanical engineer would. The next figure attempts to sum up this idea.

I think everyone can agree that there is an application development life cycle – applications have to be designed, developed, tested, and deployed. People may argue about the details, but for the purposes of this discussion we can set aside those religious debates about methodologies. All I care about here is that each stage of your development process -– whatever it is -- must include a deliberate focus on performance (or SLM), alongside other aspects of site usability.

If you want an application to achieve certain levels of service, don’t expect that to just happen on its own. You have to agree on your objectives, then design, develop, and test your application with those objectives firmly in mind, and measure and manage the application to make sure you are actually achieving your objectives. Of course, there’s nothing particularly radical or new about this idea, but the added complexity of RIA’s increases the risk of failure if we don’t actually follow these well-established best practices.

The RIA Development Process
If you want to implement a really usable Rich Internet Application, you’re probably going to need to beef up your site development processes to get everyone on the same page. That's because up to now the Web Usability (with a capital 'U') profession has concentrated on the dimension I call Clarity.

Their exclusive focus on Web page design and Information Architecture was possible only because the traditional structure of a Web applications was so constrained by the limitations of the Web browser technology they had at their disposal. When all Web applications implemented the same simple thin-client model, they did not have to worry too much about application complexity.

But as we evolve to building RIAs with a separate client-side engine, the site development process must deal with the kinds of complex issues that arise during the design of distributed applications. In the past, Web designers have not had to deal with anything as technical as this. They certainly can’t buy a book on 'Web Usability' and read about these kinds of issues.

Confirming this perspective, I recently heard Scott Dietzen, President and CTO of Zimbra speaking about Ajax at an MIT/Stanford VLAB meeting. To illustrate his statements that 'Ajax is still hard' and 'You can't crank out a good Ajax application quickly' he revealed that Zimbra has had to interview 40 JavaScript developers for every one they have hired. This is telling evidence of the difference between traditional Web applications and RIAs.

And you really do have to work hard to get everyone on the same page, because typically the people in the various participating departments usually don’t have a clear picture of just how much it takes to build a successful Web application.

As illustrated here, they naturally focus first on their own problems and challenges. The graphic is meant to be a bit tongue in cheek, but it does reflect the political realities of application development.

Several groups contribute to the success of any site, and each group thinks that their contribution is the most important! Apart maybe from those who work in IT, who tend to complain that they could deliver much better service levels if only the developers would hand them more robust and stable applications to manage.

These attitudes are not just the result of people having an inflated sense of their own importance. Most people honestly don’t understand just how much work the other players have to do.

This is already true today, and with the added compexity introduced by Rich Internet Applications everyone will have even more to think about. This is likely to increase their isolation from the other participants, unless the development process requires close cooperation.

So the first lessons your development processes must drive home are that all participants play an equally vital role in delivering a usable application, because all four dimensions of usability are equally vital to the success of the application.

The Importance of Being Agile
Building successful RIAs will also require more communication. We’ve been talking about the importance of SLM within the life cycle. But the problem with describing the software life cycle using a list activities like the one in the third illustration above (introducing the development life cycle) is that such lists tend to imply a step-by-step process –- a waterfall methodology –- in which we must complete one activity before moving on to the next. That approach to the development process is certainly not going to be an effective way to develop Rich Internet Applications.

Previously I introduced the RIA Behavior Model shown here. Consider in particular the grey boxes along the top, representing the application's design and usage environment, or context. As I wrote at the time:

A user's satisfaction with any application depends on ... how well the application design matches their needs at the time, their way of thinking, and their behavior when they are using it.

Their experience of response time depends on the combined behaviors of the client and server components of the application, which in turn depend on the application design, the underlying server infrastructure design, and of course the user's Internet connection speed. The most effective RIA will be one whose creators took into account these factors at each stage of its development life cycle, and who created the necessary management processes to ensure its success when in production.

Now imagine trying to focus on any one of the four aspects of usability without paying attention to the others. It can't be done. Application utility is tied to user behavior, which is driven by the clarity of the site, the design of the client-side engine, and the responsiveness of interactions between the client engine and the server components. And the design and implementation of those components will ultimately also determine application availability.

Since the four dimensions of usability are so intertwined, I conclude that to ensure a successful outcome, we must adopt a development process that keeps all four usability goals in focus, all the way through the life cycle. This approach is called agile software development.

Agile methods use an iterative approach to design, development and testing that keeps all the different participants ('stakeholders') involved -- marketing or business owners, Web designers, application developers, and the IT staff who understand what it takes to manage a site in production. They all have to communicate and share their different areas of expertise throughout the process.

In the Web design literature, methods similar to this have sometimes been referred to as customer-centered design. For example, that terminology is used in The Design of Sites by Doug van Duyne et al (see Chapter 1). However, Customer-Centered Design: A New Approach To Web Usability by Kreta Chandler and Karen Hyatt, while interesting (see Chapter 1), does not have much to say about the development process.

My graphic here is based on Chapter 5 of 'The Design of Sites', entitled 'Processes for Developing Customer-Centered Sites'. That book, having been published in 2003, is not about RIAs. Nor does it mention agile development per se, which is hardly surprising considering that the agile manifesto was crafted during the time when van Duyne et al were writing their manuscript.

However, it does suggest the kind of iterative approach to the development process that I believe is essential when developing RIAs. Chapter 5 concludes:

The principles and techniques of customer-centered design and iterative prototyping are embedded in every stage. Many firms have similar processes, though the stages and deliverables might have slightly different names.

Another book I have recommended previously, Usability for the Web by Tom Brinck, Darren Gergle, and Scott D. Wood, is structured around the major stages of an iterative development process. In Chapter 1 they write:

At each stage we want to cycle between refining our design and evaluating our latest refinement, iterating until we've achieved a level of usability that we're satisfied with before continuing to the next stage. Evaluation at each stage allows us to incorporate user and client feedback loops to optimize the design.

At each evaluation, we determine whether our design is adequate for continuing on. We do this by establishing benchmarks, or target usability goals.

These ideas did not emerge for the first time with the advent of the Rich Internet Application -- they are merely descriptions of good development practices. But the added complexity of the RIA model makes it essential that we actually put these kinds of methods into practice.

Managing Rich Internet Applications [6]

2006-03-25T19:14:00.000Z

This is the sixth post in a series devoted to the challenges of Service Level Management (SLM) for Rich Internet Applications (RIAs). In these applications, some processing is transferred to the Web client while some remains on the application server. Previous posts introduced the subject and the SLM topics I plan to address, reviewed RIA technologies, introduced The RIA Behavior Model, and introduced the application measurement topic.

Measuring RIA Responsiveness: Complications

To explain the challenges of measuring Rich Internet Applications, I will begin by reviewing three significant complications introduced by RIAs. From this discussion, I will draw conclusions about how measuring RIAs will differ from measuring traditional Web applications.

First Complication: Variety of Possible Behaviors
To make meaningful statements about an application's performance, you must first define what you need to measure. Typically you must measure common application usage patterns (or 'scenarios'), or patterns that are important by reason of their business value. When an application uses only a standard browser, its behavior is simple, known, and predictable, limiting the number of interesting usage scenarios to be measured.

Adding a client-side engine separates user interface actions from server requests and responses, giving application designers many more options. It allows developers to build an application that uses creative, clever ways to transfer display elements and portions of the processing to the client. The custom application behaviors encoded in the client-side engine make the result more complex and its usage less predictable than a traditional browser-based application. This increases the number of possible usage patterns, complicating the task of determining the important scenarios and measuring their performance.

Having many more design and implementation options also creates new opportunities for developers to make performance-related mistakes. They can (accidentally or deliberately) implement "chatty" client/server communication styles, which under some workload conditions may perform poorly. Even with thorough testing, some of these problems may remain undiscovered until after the application is deployed. A systematic SLM process must include measurement capabilities that provide a way to identify, investigate, and fix these types of problems.

Second Complication: Concurrent Activity
There are two reasons why it takes a while to download a Web page comprised of 50 separate content elements, no matter how fast your connection. First HTTP limits the rate at which clients can request objects from servers, then TCP limits the rate at which the data packets can deliver those objects from server to client. In particular, although the HTTP 1.1 standard allows clients to establish persistent connections with servers, RFC2616 defining the standard restricts the number of parallel connections the client (usually a browser) can use:

[Section 8.1.4] Clients that use persistent connections SHOULD limit the number of simultaneous connections that they maintain to a given server. A single-user client SHOULD NOT maintain more than 2 connections with any server or proxy. ... These guidelines are intended to improve HTTP response times and avoid congestion.

Avoiding bottlenecks is a desirable goal in all computer systems, and so the Internet protocols are designed to protect shared network and server resources, no matter how selfishly their users might behave. But flow controls may not always be needed. Consider the metering lights that permit only two cars onto the highway every 30 seconds. During the rush hour they can smooth the flow of traffic and reduce freeway congestion, but if left on at other times, they would delay drivers for no reason. Similarly, when there is plenty of available bandwidth and no server congestion, the limit of two connections per domain is just a governor that restricts the potential rate of data transfer from server to client.

Nonetheless, modern browsers (including IE) do adhere to this guideline. And it is safest to do so, because DoS logic in some proxy servers might reject connections that do not obey the RFC2616 standard. For a lively discussion of IE behavior and the meaning of 'SHOULD' in the RFC, see this msdn blog, which also points out a relatively simple technique any Web site designer can use to circumvent this limitation, if they feel it is important enough:

Savvy web developers can take this connection limit into account and deliver their data from multiple domains, since the browser will open up to two connections per domain.

Although this solution may be fast enough for many applications, most Ajax developers are looking for clever ways to make the client even more responsive. And because Ajax offers almost unlimited application design possibilities, today there is little agreement on the best ways to achieve that goal. This debate was well summarized by Jep Castelein in AJAX Latency problems: myth or reality?, a discussion that includes the important reminder that 'IE will have a maximum of 2 simultaneous connections (per domain, actually -- C.L.), whether you use XMLHttpRequest or not'. In other words, Ajax implementations are not above the law of HTTP.

Even so, a primary objective of many RIA designs is to work around the two-connection obstacle to improve responsiveness for the user. The resulting implementations will use many different techniques to communicate with one or more servers. As one data point, consider the opinion of Drew McClellan, who surely qualifies as an expert in developing Web applications. In his tutorial about JavaScript and the XMLHttpRequest object -- Very Dynamic Web Interfaces -- Drew concludes that the real challenge here is not figuring out how to make the code work but thinking of interesting ways in which it can be utilized.

Such freedom to improvise inevitably complicates measurement -- especially when requests originating from what appear to be separate units of work on a client are really all part of a single logical application. It is easy to sit on a server and measure the traffic, or the demands placed on various server-side resources. And this kind of measurement has its uses, especially when load testing or investigating bottlenecks. But the biggest challenge when measuring applications is correlating those seemingly separate pieces of information with a particular application activity, task, or phase. The more complex the client/server relationship, especially when it involves concurrent interactions, the trickier it becomes for measurement and analysis tools to perform that correlation properly.

Third Complication: Asynchronous and Synchronous Communications
An earlier post discussed RIA technologies. It described how Flash and Ajax use client-side engines that can be programmed to communicate asynchronously with server(s), independent of a user's actions. This notion is captured in name 'Ajax', in which the initial 'A' stands for 'Asynchronous', and Figure 2 in that post was taken from Jesse James Garrett's seminal article on Ajax. Although Garrett's figure does show how an engine enables asynchronous application behaviors, and differs from a traditional Web app, it does not illustrate the full range of possibilities, which I discussed further in RIA post #4. In general, a user action within a Rich Internet Application can trigger zero, one, or many server requests.

Also, most discussions of RIA architecture or technology focus on how asynchronous application behaviors can improve usability. But they don't question the fact that communication between browser and server is still synchronous. That is, communication is always initiated by the browser (or the client-side engine operating as an extension of the browser), and follows the standard synchronous HTTP request and response protocol. But in a recent blog post and presentation, Alex Russell of dojo proposed the name Comet for a collection of clever techniques that exploit HTTP persistent connections to implement a 'push' model of communication.

He used an adaptation of Garrett's original Ajax figure to show how Comet extends the Ajax model; this smaller version comes from brainblog:

Using the Comet techniques, a server uses long-lived persistent connections to send updates to many clients, without even receiving a request (a 'poll') from the client. According to Russell:

As is illustrated above, Comet applications can deliver data to the client at any time, not only in response to user input. The data is delivered over a single, previously-opened connection. This approach reduces the latency for data delivery significantly.

The architecture relies on a view of data which is event driven on both sides of the HTTP connection. Engineers familiar with SOA or message oriented middleware will find this diagram to be amazingly familiar. The only substantive change is that the endpoint is the browser.

While Comet is similar to Ajax in that it's asynchronous, applications that implement the Comet style can communicate state changes with almost negligible latency. This makes it suitable for many types of monitoring and multi-user collaboration applications which would otherwise be difficult or impossible to handle in a browser without plugins.

Like Ajax, Comet is not a new technology, but a new name for some clever ways to implement an old communication style using standard Web protocols. But as happened with Ajax in 2005, the new name has triggered a lot of interest among developers -- for evidence, just search on 'Ajax', 'Comet', and 'Web' (the last term should eliminate most links to Greek legends, soccer legends, and legendary cleaning products). Especially useful are Russell's slides from his talk (Comet: Low Latency Data For Browsers) at the recent O'Reilly ETech Conference, and Phil Windley's excellent write-up of the talk.

The complexity of a push architecture is justified only for applications that manage highly volatile shared data. But when it is implemented, it creates an entirely new set of measurement challenges. If information pushed to the client does help the user to work faster, its benefits will be reflected in normal measurements of application responsiveness. But you can't use normal responsiveness metrics to evaluate a server-initiated activity that simply updates information a user is already working with.

New metrics must be devised. Depending on the application, we may need to measure the currency or staleness of information available to the user, or maybe the percentage of times a user's action is invalidated by a newly received context update from the server. This kind of "hiccup" metric is conceptually similar to the frequency of rebuffering incidents, one measure of streaming quality. Server capacity will also be a major issue requiring careful design and load testing. These are new SLM challenges facing anyone who decides to implement the Comet approach.

Measuring RIA Responsiveness: Conclusions

While discussing the three major complications introduced by RIAs, I have already noted some consequences: measurements may become harder to specify, more difficult to correlate, and may even require the invention of new metrics. But I have been saving the punch-line until the end. I will now discuss my three most important and far-reaching conclusions about measuring RIAs, each of which involves a significant change from the way Web applications are measured today. These changes deal with the issues of what, where, and how to measure.

RIAs: What to Measure?
The short answer? Not individual Web pages. First and foremost, because the asynchronous aspects of the RIA model undermine two fundamental assumptions about Web applications and what to measure:

We can no longer think of a Web application as comprising a series of Web pages.
We can no longer assume that the time it takes to complete a Web page download corresponds to something a user perceives as important.

In fact, when a client-side engine can be programmed to continually download new content, or a server-side engine can keep pushing new content to the browser over a connectionn that never closes, some page downloads may never complete.

Therefore to report useful measurements of the user experience of response times, instead of relying on the definition of physical Web pages to drive the subdivision of application response times, we must break the application into what we might call 'logical pages'. To do this, a measurement tool must recognize meaningful application milestones or markers that signal logical boundaries of interest for reporting, and thus subdivide the application so that we can identify and report response times by logical page.

Because (as I noted earlier) it is usually hard to correlate seemingly separate pieces of measurement data after the fact, I conclude that these milestones will have to be identified before measurement takes place. They could be key events or downloads that always occur naturally within the application. Or they could require proactive instrumentation, for example using little downloadable content elements that developers deliberately imbed at logical boundaries in the application's flow, to enable its subsequent measurement.

The former method places the burden on those setting up measurements to first identify application-specific milestones. The latter frees the measurement tool from the need to know anything about the application, but places the burden on application developers to instrument their code by inserting markers at key points.

Comparing Apples and Oranges?
Second, when setting out to measure a RIA, you must think carefully about the purpose of the measurement, especially if the RIA is replacing a traditional Web application, or being compared with one. You must not let the traditional Web approach to a business task determine what measurements are taken. Aleks Šušnjar (see post [2] in this series) provided the following insights:

We can't compare apples and oranges by measuring either an apple or an orange. We need to approach measurement as if we were comparing two applications built by different developers -- one a traditional Webapp and the other a client/server app with a completely different UI.

In this situation, we cannot measure only at the level of single Web pages or server interactions. Finding that it takes so many milliseconds to get a server response for request-type X is meaningless if that client/server interaction does not occur in both versions of the application.

In my experience, a typical example concerned how long it took to upload or download a document. But those metrics were sometimes irrelevant, depending on the application context. So to make really useful performance comparisons, we had to approach the problem at a higher level -- for example, 'how long does it take to move a document from folder A to folder B?' In a traditional Web app that task would likely require multiple clicks on separate pages, whereas with an RIA/Ajax implementation, we could do it with a single drag and drop.

So to make valid comparisons, we had to measure and compare the net effect of two rather different aspects of performance -- one concerning only the computer (how many operations of type X can a machine perform per hour), the other involving human performance (how many documents can an employee move per hour). But both affected the overall conclusion. Generalizing, I would say that:

The server that has to generate additional HTML for multiple responses in a traditional Web app will likely use many more processor cycles than the one supporting an RIA/Ajax implementation, where all the user interactions are handled on the client and the server just has to provide a service at the end.
If the designer takes care to avoid 'chatty' client/server communications, network utilization will probably also be significantly lower in the second case, further improving server performance.
Finally, if the client-side interface is well designed, a RIA should allow users to think and work faster.
In the final analysis, all these components of application performance must be weighed.

Aleks' insights come from his own experience developing a Rich Internet Application -- for more details see his Wikipedia page about RIA and AJAX.

RIAs: Where to Measure?
You might think that to measure a user's experience of responsiveness, you would have to use a tool that collects measurements from the user's workstation, or at least from a measurement computer that is programmed to generate synthetic actions that imitate the behavior of a typical user. Surprisingly, this is not actually the case for traditional Web applications.

While synthetic measurements require computers to mimic both a user's actions and their geographical location, software that collects passive measurements of real user actions can in fact reside either on the client machine or on a machine that is close to the server, usually just behind the firewall -- just so long as that machine can observe the flow of network traffic at the TCP and HTTP levels. Because of the synchronous and predictable nature of these protocols, a measurement tool that can read and interpret the flow of packets can actually deduce the user's experience of response time by tracking HTTP message traffic and the lower-level timings of TCP data packets and (crucially) TCP acknowledgements.

Such a tool is called a packet sniffer, or protocol analyzer. Packet sniffing has a bad name in some quarters, being associated with malicious snooping by hackers. But in the right hands, it is a legitimate analysis technique used by some Web measurement tools to deduce client-side performance without actually installing any components, hardware or software, anywhere near the users.

Unfortunately for these tools, the growth of RIAs will make it progressively more difficult to exploit this clever approach. The RIA Behavior Reference Model (this figure) I introduced previously makes it clear that RIAs -- even without the further complications introduced by a Comet push architecture severely limit the power of the packet sniffing approach. That's because we can no longer characterize response time as the time to complete the synchronous round trip of:

Click(C) => Browser(B) => Request(Q) => Server(S) => Response(R) => Display(D)

Instead the client-side engine in the RIA architecture breaks apart this cycle into two separate cycles operating asynchronously:

The user-client cycle: Click(C) => Engine(E) => Display(D) -- [CED, for short]
The client-server cycle: Request(Q) => Server(S) => Response(R) -- [QSR, for short]

Another way of describing these cycles might be as 'foreground' (CED) and 'background' (QSR). Both cycles are important, because neither stands alone; it is their relationship that defines application behavior. But that relationship depends only on the application design, which cannot (in general) be inferred by a measurement tool, especially one that can observe only one of the two cycles.

I conclude therefore that to measure RIAs, tools will have to reside on the client machine, where they can see both the level of responsiveness experienced by the browser (the QSR cycle) and the user's experience of responsiveness (the CED cycle). Granted, QSR cycles can still be monitored by the traditional packet sniffing methods, but tracking them will permit only limited inferences to be made about the CED cycle, which is separately managed by the client-side engine.

One might imagine that tracking the times of certain previously identified 'marker' objects within the QSR stream could solve this problem, especially since I already concluded (above) that marker objects will be needed to delimit the logical pages of RIAs. But in order to be able to draw conclusions about CED times by seeing those markers in the QSR stream, a measurement tool must impose a lot of constraints on the design of the client-side engine. An engine that implemented truly asynchronous behaviors (such as anticipatory buffering) would make it difficult or impossible to assess the user's actual experience without a measurement presence on the client side to observe the CED cycle.

Either that, or the marker objects would themselves need to be active scripts that triggered timing events that were somehow transmitted to the server (in a manner similar to ARM), rather than simply being passive milestones. But once again, this approach is tantamount to placing a measurement capability on the client. (Indeed, in the context of RIAs, dynamically distributing little measurement scripts that function as extensions to the client-side engine would be a natural approach). I therefore conclude that an approach comprising only passive listening on the server side will be insufficient to measure RIA responsiveness.

RIAs: How to Measure?
We have seen that RIAs will affect where a passive measurement tool can be used. Active measurement tools, because their modus operandi is to simulate the user's behavior, are not affected by this issue -- since they mimic the client, they naturally reside there. For these measurement tools, the issue raised by RIAs is how closely a user's behavior needs to be emulated.

User Actions: First, note that RIAs can generate back-end traffic in response to any user action, and not only when the user clicks. For example, the Google maps application can trigger preloading of adjacent map segments based on the direction of a user's cursor movement within the map display. Therefore to measure RIA responsiveness, an active measurement tool must simulate the user's actions, not the engine's actions. This further explains why active tools must reside at the client.

In other words, using the terminology introduced earlier, active measurement tools must drive the CED cycle, not the QSR cycle. The former involves driving the client-side engine to obtain its backend behavior; the latter would require the tool user to supply a complete script of the engine's behavior to the active measurement tool. The latter involves more work and is inherently more difficult and mistake prone, and therefore much less useful.

Think Times: Second, an active measurement tool must properly reflect then fact that a client-side engine may be doing useful work in the background, while a user is reading the previous response or deciding what to do next. It may, for example, be prefetching content in anticipation of the user's next action. Therefore the time a user devotes to these activities -- jointly represented as 'think time' in the RIA Behavior Model -- may affect their perception of the application's responsiveness.

For passive measurements of real user traffic, this is not a problem, because their measurement data always includes the effects of the users' actual think times. But for traditional Web apps, synthetic measurement tools have not needed to simulate think times by pausing between simulated actions, because introducing such delays would not have altered the result. When measuring an RIA however, setting think time to zero (as is typically done today) could have the effect of eliminating or minimizing potential background preloading activity, thus maximizing the perceived response times of later user actions.

And because the engine's behavior during think time varies by application, a measurement tool cannot simply measure content download times then introduce simulated think times during the reporting phase. Combining individual component measurements after the fact to produce a realistic estimate of user experience would be like trying to construct a PERT chart for a volatile project when you are not sure you know all the tasks and also cannot be sure about all their interdependencies -- in other words, impossible.

While people familiar with the project could probably construct the chart and draw conclusions about the project's duration, a general-purpose tool cannot. But in software performance and analysis work, the most difficult and error-prone aspect is combining low level measurements to draw conclusions about user-related metrics like transaction response time. So most users of application measurement tools want to be told the bottom line, namely the user experience, not just the underlying component times.

Therefore I conclude that to reflect a user's experience, an active measurement tool will have to simulate user think times during the measurement process. Using realistic think times (as the best load testing tools already do today) will produce the most realistic measure of the response times a user perceives during a particular application use case.

Summary
Since this has been a long post I will now summarize my conclusions, with links back to the discussion behind each:

The variety of possible RIA behaviors creates new opportunities for developers to make performance-related mistakes, requiring more systematic approaches to measurement.
Concurrent client/server interactions make it difficult for measurement and analysis tools to correlate seemingly separate pieces of data with a particular application activity.
RIA push models (like 'Comet') will require the invention of new metrics to measure effectiveness.
Milestones must be specified in advance to allow measurement tools to group RIA measurements into 'logical pages' or tasks.
To compare the performance of a RIA with a traditional Webapp, you must measure equally important activities within each.
Passive monitoring of server requests will be insufficient to determine a user's perception of RIA responsiveness.
Active measurement tools must simulate user actions, not just engine actions, to measure RIA responsiveness.
Active measurement tools must simulate user think times during the measurement process, to reflect a user's experience accurately.

Next ...
This completes (for now) my analysis of the challenges of measuring RIAs; any further ideas will have to wait until future posts. Next I will consider how the introduction of RIAs may affect SLM processes that were designed and fine-tuned to manage traditional Web applications.

Managing Rich Internet Applications [5]

2006-03-17T08:32:00.000Z

This is the fifth post in a series devoted to the challenges of Service Level Management (SLM) for Rich Internet Applications (RIAs). In these applications, some processing is transferred to the Web client while some remains on the application server. Previous posts introduced the subject and the SLM topics I plan to address, reviewed RIA technologies, and introduced The RIA Behavior Model.

Measuring RIA Responsiveness: Introduction

Having been sidetracked by my interesting but futile search for the origin of the saying 'you can't manage what you can't measure', I am now almost ready to examine some of the challenges of measuring Rich Internet Applications. But first I must spell out the motivations for measuring an application, and explain why the choice of an active or passive measurement methodology is of only secondary importance to this discussion.

Why Measure Applications?
Reasons for measuring application performance fall into two classes, one client-centric, the other server-centric. They are:

MEASUREMENT: Determining how quickly the users can achieve their goals:
- (a) Verifying that the application meets its service level objectives
- (b) Detecting abnormal application behavior (typically, slow response)
- (c) Identifying bottlenecks in application components
- (d) Comparing builds, versions, or releases of an application
- (e) Comparing two applications (e.g. our app vs. competitor's version)
LOAD TESTING: Determining how a system behaves under increasing load:
- (a) Verifying that the system will be able to handle the projected traffic
- (b) Determining how many users a given server environment can handle
- (c) Predicting bottleneck components as workload levels grow
- (d) Comparing builds, versions, or releases of an application
- (e) Identifying components that fail after extended use

As I have indicated, activities in the first class are conventionally called 'measurement', while those in the second are referred to as 'load testing', or simply 'testing'. So although both classes require us to address many common measurement problems, I will simplify my description of these problems by using the conventional terminology to distinguish the two classes of activties.

Active vs. Passive Measurement
A few years ago I edited a version of the Network Measurement FAQ originally produced by a CAIDA working group, for publication by CMG. The online FAQ seems to have gone into hiding, but you can still download the CMG paper, Fundamentals of Internet Measurement: A Tutorial. Although it is about network measurement, a couple of paragraphs introduce the concepts of active and passive measurements:

Passive measurements are carried out by observing normal network traffic, so they do not perturb the network. They are commonly used to measure traffic flows, i.e. counting the number of packets and bytes travelling through routers or links between specified sources and destinations.

Active measurements, on the other hand, are performed by sending test traffic into the network. For example, one might measure a network's maximum carrying capacity by sending packets through it and increasing the sending rate until the network is saturated. Clearly one needs to be aware that active measurements impose extra traffic onto a network and can distort its behavior in the process, thereby affecting measurement results.

When measuring Web applications, similar definitions and concerns apply, except at the level of application traffic and the server infrastuctures that deliver applications. While load testing requires an active measurement approach, application responsiveness can be measured using either active or passive methods.

However, no matter which is used, the passive and active measurement approaches differ only in the way application traffic is generated -- both still require mechanisms to measure that traffic. Passive measurements must capture the behavior and experience of real application users, while active measurements must do the same for synthetic users, that is, computers mimicing the behavior of real users.

This is fortunate, because to discuss the pros and cons of real and synthetic monitoring would require a major blog post. In fact I gave a short presentation on that topic at Interop last year, which would provide the structure required. But for now I will simply note that both real and synthetic application users have to be measured, and RIAs present many common measurement challenges, which I will be describing in my next post.

Deep Thoughts on Management

2006-03-14T21:48:00.000Z

Since I am writing a series of posts about managing Rich Internet Applications, and working on a post about the difficulties of measuring them, I thought I should begin with the popular management aphorism that you can't manage what you can't measure. Well, was that ever a diversion! Everyone is familiar with this saying, but interestingly, despite a ton of digging on the Web, the precise origin of this saying remains obscure (at least, to me).

Perhaps because of my training in statistics, I had always assumed it was just a simplification of a famous statement by Lord Kelvin, often cited by statisticians:

When you can measure what you are speaking about and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind. It may be the beginning of knowledge, but you have scarcely advanced to the stage of science.

Unfortunately, there does not seem to be a lot of support for my assumption. Instead, my Web searches have revealed that the statement 'you can't manage what you can't measure' is most often attributed either to the pioneer of quality control, W. Edwards Deming, or to 'the father of modern management', Peter Drucker. Moreover, neither of these popular attributions seems to be correct.

In Deming's case, one of his most well-known statements is almost the exact opposite, namely that management must take account of many things that cannot be measured, and that running a company on visible figures alone was one of the seven deadly diseases of management. All the same, I'm sure Deming, given his interest in quality control, would have also agreed that things which can be measured should be measured, And all SLM processes are based on measurements.

Attribution to Peter Drucker also seems to be a misconception, since it does not appear either in the wikiquote list of quotes from Drucker, or in other extensive lists of his well-known sayings. But doing this research reminded me of the many insightful statements Drucker actually did make, including:

Management is doing things right; leadership is doing the right things.
There is nothing so useless as doing efficiently that which should not be done at all.
A manager's task is to make the strengths of people effective and their weakness irrelevant--and that applies fully as much to the manager's boss as it applies to the manager's subordinates.
Management is so much more than exercising rank and privilege, ... it is much more than "making deals." Management affects people and their lives.
Rank does not confer privilege or give power. It imposes responsibility.
Executives owe it to the organization and to their fellow workers not to tolerate nonperforming individuals in important jobs.
Most of what we call management consists of making it difficult for people to get their work done.
The most efficient way to produce anything is to bring together under one management as many as possible of the activities needed to turn out the product.
Increasingly, politics is not about "who gets what, when, how" but about values, each of them considered to be absolute. Politics is about "the right to life"...It is about the environment. It is about gaining equality for groups alleged to be oppressed...None of these issues is economic. All are fundamentally moral.
What's absolutely unforgivable is the financial benefit top management people get for laying off people. There is no excuse for it. No justification. This is morally and socially unforgivable, and we will pay a heavy price for it.
Unless commitment is made, there are only promises and hopes... but no plans.
The best way to predict the future is to create it.

This post is an example of what happens when I do research on the Web -- I find a lot of interesting information I was not even looking for. So this post became a diversion about the management insights of Edward Deming and Peter Drucker, and in my next one I will get back to the subject of measuring RIAs.

Managing Rich Internet Applications [4]

2006-03-10T05:25:00.000Z

This is the fourth post in a series devoted to the challenges of Service Level Management (SLM) for Rich Internet Applications (RIAs). In these applications, some processing is transferred to the Web client while some remains on the application server. Previous posts introduced the subject, listed topics I plan to address, and reviewed RIA technologies.

The RIA Behavior Model

In an earlier post I discussed the idea of using a reference model to establish a shared frame of reference, or conceptual framework, to structure subsequent discussions of a subject. Today I introduce a new reference model -- The RIA Behavior Model -- illustrated by the diagram below.

I intend this model to represent the principal elements of the behavior of RIAs, elements that must be considered in any discussion of RIA performance and management. Note however that I am not attempting to address the complex human forces that determine perceptual, cognitive, and motor behaviors. I am merely seeking to represent a few generalized behavioral outcomes that are relevant in the context of an interaction between a human user and a Rich Internet Application.

As you can see, this model embraces more concepts than the two figures created by Jesse Garrett to introduce Ajax, which I included in my previous post. Those figures are ideal for explaining the differences between traditional Web applications and RIAs. But to discuss how the combination of a Rich Internet Application and a user will actually behave, I have included several more elements that interact to determine behavior, which I will now describe.

At the highest level, the model comprises three major aspects (indicated by the color coding in the figure), each of which influences application performance:

The application's design and usage environment, or context (upper row, grey)
The user's expectations and behavior (lower left, blue)
The application's behavior when used (lower right, yellow)

Browser/Server Interaction
If we consider a Web browser to be the simplest form of client engine, then the solid black arrows trace the flow of a traditional Web page download. The user clicks on a link in their browser, the browser sends requests to one or more servers. Servers respond to client requests, and when there is enough of the requested content on the client (in the browser cache), the browser renders it ('paints' the screen), and the user can view it. The user's experience of response time is the elapsed time of the entire process, 'from click to paint'.

Even a single Web page download will normally involve many round trips between client (browser) and server, because most Web pages are an assemblage of content elements such as CSS files, script files, and embedded images, each of which is separately downloaded by the browser.

In the traditional synchronous Web application, illustrated in the upper half of Garrett's Figure 2, this process is repeated several times. Because applications usually require an exchange of information, at least one of the requests the browser sends to the server will normally be an HTTP POST (as opposed to the much more common HTTP GET request), to upload some data a user has entered into a form. Consider, for example, shopping at amazon.com as a return visitor. At minimum, even if the application recognizes you from a cookie, you must enter your password again to confirm your identity. But after that, the site already has all your personal information, and you can complete your transaction just by clicking on the right buttons on each page as it is presented to you.

Server-Side Elements
We are all familiar with this kind of Web application and its behavior. But unless you are responsible for site management or performance, you may be less aware of some of the other server-side elements of the model. Servers must field requests concurrently from many users. No matter how powerful the server, every concurrent user consumes their small share of the server's resources: memory, processor, and database.

Web servers can respond rapidly to stateless requests for information from many concurrent users, making catalog browsing a relatively fast and efficient activity. But when a user's action (such as clicking the 'add item to shopping cart' button) requires the server to update something, more of those server resources are consumed. So the number of concurrent transactions -- server interactions that update a customer's stored information -- plays a vital role in determining server performance.

In the model, the grey arrows and the boxes labeled Users and Transactions represent the fact that server performance is strongly influenced by these two concurrency factors. Servers typically perform uniformly well up to a certain concurrency level, but above that level ('the knee of the curve') transaction performance quickly degrades, as one of the underlying resources becomes a bottleneck. Because of this characteristic, seemingly small design changes in an application or in the infrastructure serving the application may, if they extend the duration of transactions, have a significant effect on the user's experience of response time.

Adding a Client-Side Engine
Adding a client-side engine does not prevent an application from implementing the traditional synchronous design described above. But RIAs aim to improve the user's experience, and a client-side engine allows the application designer to consider many additional possibilities, such as:

Prefetching of content to client
Lazy loading of content
Just In Time fetching of content
Client-side validation of user input
Client-side only responses to user input
Batching of server inputs on the client
Offloading of server function to client machines

If any of these techniques are employed, the resulting application will inevitably be more complex than the traditional synchronous application. The challenge of SLM is to ensure that a more complex design actually does produce a more responsive user experience, because -- despite the optimistic claims being made for Ajax and Flash -- there are no guarantees.

Improving the User Experience
For example, a common method of accelerating client-side response is to anticipate the user's next action(s) and program the engine to preload (or prefetch) some content 'in the background', while the user is thinking. Depending on their think time, when the user clicks, part or all of the desired response can be already available on the client. This technique has long been used in client/server systems to improve client responsiveness; a version (called Link Prefetching) is implemented by Mozilla browsers.

Preloading will certainly create a more responsive experience -- if the user actually follows the anticipated path through the application. But what if they don't? Then the client engine has made extra server requests that turned out to be unnecessary, and it still has to react to the user's input with yet more server requests. So the design has placed extra load on the servers, for no benefit.

The extra load serving these 'background' requests, on behalf of what may be hundreds or even thousands of clients running the preloading application, can slow down a server's responses to requests that users are actually waiting for. Slower responses lengthen transaction times, which drives up the number of concurrent users, clogging up the servers even more, and further slowing down responses. It's a vicious circle.

Even if the application serves some users quickly, those whose usage patterns do not match the profile the application designer had in mind will probably not receive such good service, and (in the worst case) may abandon transactions before completing them. Apart from the lost opportunity to serve a customer, abandonment usually also wastes scarce server resources, because the allocations earmarked for now-abandoned transactions languish unused until finally freed by some type of timeout mechanism.

People who design and test back-end systems already know that behavioral variables like user think-time distributions and abandonment rates per page have a significant influence on the capacity and responsiveness of servers under load. Today, RIAs (as indicated by the dotted lines in the diagram) give application designers the flexibility to create application designs that attempt to take account of such behavioral variables. But a consequence is that RIAs also magnify the risk of failure should the designer miscalculate: a simple design applied well beats a clever design misapplied.

The Application Environment
This brings us to the crucial importance of the design and usage environment, represented by the grey boxes in the model. A user's satisfaction with any application depends on their usage context and environment; in other words, how well the application design matches their needs at the time, their way of thinking, and their behavior when they are using it.

Their experience of response time depends on the combined behaviors of the client and server components of the application, which in turn depend on the application design, the underlying server infrastructure design, and of course the user's Internet connection speed. The most effective RIA will be one whose creators took into account these factors at each stage of its development life cycle, and who created the necessary management processes to ensure its success when in production.

Future Posts
Using The RIA Behavior Model as one starting point, future posts will discuss how to build robust and responsive RIAs, and explore some of the technical and management challenges they pose, especially in the area of response time measurement. If I can organize my thoughts sufficiently to keep the writer's block at bay, I expect my next post to focus on that topic.

Managing Rich Internet Applications [3]

2006-03-07T04:56:00.000Z

This is the third post in a series devoted to the challenges of Service Level Management (SLM) for Rich Internet Applications (RIAs). In these applications, some processing is transferred to the Web client while some remains on the application server. Previous posts introduced the subject, and listed the topics I plan to address.

RIA Technologies
Before diving into the management issues posed by Rich Internet Applications, I will introduce the two principal technologies used to implement RIAs -- Ajax and Flash -- and provide a few links I have found useful.

I am going to begin with Ajax, because the seminal article defining the term "Ajax," Jesse James Garrett provides such a clear introduction to the subject. After explaining that ...

Ajax isn’t a technology. It’s really several technologies, each flourishing in its own right, coming together in powerful new ways. Ajax incorporates:
standards-based presentation using XHTML and CSS;
dynamic display and interaction using the Document Object Model;
data interchange and manipulation using XML and XSLT;
asynchronous data retrieval using XMLHttpRequest;
and JavaScript binding everything together.

... he uses two figures that reveal the essential differences between a classic Web application and one implemented using Ajax:

The classic web application model works like this: Most user actions in the interface trigger an HTTP request back to a web server. The server does some processing — retrieving data, crunching numbers, talking to various legacy systems — and then returns an HTML page to the client... This approach makes a lot of technical sense, but it doesn’t make for a great user experience. While the server is doing its thing, what’s the user doing? That’s right, waiting. And at every step in a task, the user waits some more.

Obviously, if we were designing the Web from scratch for applications, we wouldn’t make users wait around. Once an interface is loaded, why should the user interaction come to a halt every time the application needs something from the server? In fact, why should the user see the application go to the server at all?

Figure 1 illustrates how Ajax is different:

Figure 1: The traditional model for web applications (left) compared to the Ajax model (right).

While this diagram refers to Ajax technology, its structure describes RIAs in general. Garrett explains that the RIA model:

... eliminates the start-stop-start-stop nature of interaction on the Web by introducing an intermediary — an Ajax engine — between the user and the server. ... At the start of the session, the browser loads an Ajax engine — written in JavaScript and usually tucked away in a hidden frame. This engine is responsible for both rendering the interface the user sees and communicating with the server on the user’s behalf".

Figure 2 (below) illustrates how the addition of a client-side engine ... "allows the user’s interaction with the application to happen asynchronously — independent of communication with the server".

Figure 2: The synchronous interaction pattern of a traditional web application (top) compared with the asynchronous pattern of an Ajax application (bottom)

In the best case, a client-side engine can mean that users spend less time waiting for the server to respond. Like all writers making the case for Ajax and RIAs, Garrett assumes that this architecture guarantees a more responsive user experience -- but the reality is more complicated. In practice, an RIA's responsiveness will depend on several factors that I will be exploring in this series of posts.

For more detailed discussion of the history of RIAs and Ajax, I recommend Aleksandar Šušnjar's Wikipedia page about RIA and AJAX. For more technical details, see these Sitepoint articles:

What about Flash?

You may feel I should have started with Flash, because it came first, but opinion is divided on that point. It is true that Macromedia announced their Flash MX product line in 2002 (see Developing Rich Internet Applications with Macromedia MX for a good summary), whereas the term Ajax was coined only in February 2005. However, the underlying RIA techniques have been in use for much longer. For example, on his Wikipedia page Aleks cites the pioneering work of the now defunct company, Desktop.com. In 1999, Dave Winer wrote in his blog:

Want a vision of where the Web is headed? Check out Desktop.Com. Launched in beta a couple of weeks ago, this stunning site changed my point of view on what can be accomplished with JavaScript, ActiveX and whatever other kinds of mysterious code-doo-dads they're using... Desktop.Com says the Web is a desktop, just like the desktops on the Mac and Windows. Icons down the left edge of the browser window, menus at the top of the window, double-click to open a directory, double-click to edit a file. -- Dave Winer

If you need to compare Ajax and Flash, here are four links you may find interesting:

Any search engine will turn up plenty more material on this subject, and I will return to it when discussing aspects of RIA measurement and management.

You can download many reports and papers about Flash and RIAs from the Adobe (formerly Macromedia) Web site -- for example, these. Not included in that list is one of the most readable introductions to Rich Internet Applications, a 2003 paper sponsored by Macromedia and Intel and written by Joshua Duhl of IDC, an excellent writer who I once worked with (in the early 90's) at ONTOS, a Boston-based Object DB company. Another former colleague from Boston, Alan Sarasohn, used to claim that the computer industry is run by just 300 people, but they keep moving around. I'm starting to believe him!

Managing Rich Internet Applications [2]

2006-03-02T08:20:00.000Z

This is the second post in a series devoted to the challenges of Service Level Management (SLM) for Rich Internet Applications (RIAs). In these applications, some processing is transferred to the Web client while some remains on the application server.

In the first post in this series, I introduced the concept of a Rich Internet Application, and asserted that anyone planning to deploy RIA technology must be prepared to invest more in design, development, testing and management to make it successful. By that I meant that deploying an RIA successfully will demand more resources -- skills, tools, process, time, and (of course) money -- than would be required to deploy a more traditional version of a Web-based application in which all application logic is executed on the server.

I have reached this conclusion after studying this subject for the last three months, in the light of my prior experience in the field of software performance management. Factors leading to this conclusion include:

the nature of much of the technology being promoted for building RIAs today (primitive)
the stage of evolution of this new application architecture (early)
the toolsets available to support RIA developers (incomplete), and
the experiences of others developing RIAs (hard work).

This comment by Todd Spangler in Baseline Magazine neatly sums up the situation today:

Creating an Ajax application from scratch is like having to build a brick wall but first having to figure out how to create the bricks.

This reminds me of the days when I used to program in IBM Assembler -- funny how life in the world of computing seems to repeat itself every few years! (And I'll have more to say about that later). But while this is all fascinating stuff, it is not my primary focus; it is just the starting point for the SLM topics I want to discuss. So although I will devote some space to reviewing the state of RIA technology, I will try to keep those posts concise and provide useful links. If you want more, a few searches will turn up tons of reference material.

The questions I plan to explore in more depth in this series are:

Why are RIAs so much more difficult to design, develop, and manage?
How do you measure an RIA user's experience of response time?
How do you test whether an RIA will perform acceptably in the production environment?
How do you break apart the time it takes to complete a business transaction using an RIA into meaningful components?
How do you monitor the performance of an RIA in the production environment, and alert when its behavior is abnormal?
What kinds of development and systems management processes will maximize the chances of implementing an RIA successfully?

Before I get into any of these details, I'd like to acknowledge the contributions of two people who have helped shed light on these issues. This first is Victor Pavlov, a principal engineer and colleague at Keynote, who knows Web technology inside out, and can figure out how to measure anything. I am lucky that when I am noodling on a tricky question, I can sometimes run into Victor at the coffee machine.

The second is Aleksandar Šušnjar, who I met online after reading his contributions to the discussion of RIAs in Wikipedia, where he shared insights gained from his experience building a product first released in 2002 by his company, Hummingbird. This product, DM Webtop (later renamed DM Web Server) is a component of their Enterprise Document Management suite. In delivering desktop capabilities via the Web, it clearly predated much of today's thinking about the purpose of RIAs and how to build them.

Naturally, Aleks also discovered some potential pitfalls an RIA designer might run into, which I will be mentioning in later posts. Regrettably, many of his insightful contributions to Wikipedia (which can still be found in its history pages) were subsequently removed by other contributors who lacked either his intimate knowledge of the technology or his historical perspective. Rather than waging Wikipedia edit wars, which for a hot topic can continue interminably, he has simply posted his own page about RIA and AJAX.

I am not surprised by these events, because I have found that the computing world tends to attract people who have little sense of history. This explains why it is forever reinventing the wheel, repeating yesterday's mistakes, and tripping over previously solved problems. Rich Internet Applications are a case in point. Mainframe (thin client) computing was replaced by the client/server (fat client) model, which in turn was displaced by the Web-based (thin client) applications. Now the emergence of RIAs signals a return to the fat client model again. The only difference between the client/server and RIA models is that instead of Sneakernets and static installation protocols, Internet and Web technologies now provide an almost universally available platform for distributing function to client desktops dynamically. But many of the over-optimistic claims being made for RIA technology today mirror those popular 15-20 years ago, when everyone was jumping on the client/server bandwagon and predicting the imminent demise of the mainframe -- which, by the way, never happened.

So I'll end by reminding you of the famous saying of philosopher George Santayana: Those who cannot remember the past are condemned to repeat it. One of my goals in this series of posts about RIAs is to keep you from that fate, so I'm glad to be collaborating with people like Victor and Aleks whose common sense is grounded in a strong sense of history!

Managing Rich Internet Applications [1]

2006-03-01T07:30:00.000Z

After a few months to collect my thoughts, I have at last conquered my writer's block. Today I begin a series of posts on the topic of Managing Rich Internet Applications. The term Rich Internet Application, which I will often abbreviate to RIA for convenience, refers a Web-based application that Wikipedia defines as "a cross between web applications and traditional desktop applications, transferring some of the processing to a web client and keeping (some of) the processing on the application server". RIAs are most often implemented using either the Flash or JavaScript technologies, although the client-side processing could also be coded as a Java application or applet.

Wikipedia provides a useful, though brief, introduction to this subject; I will provide more references in future posts. Greatly complicating any coherent analysis of RIA technology is the massive amount of hype surrounding both Web 2.0 (a superset of RIA) and Ajax (a subset of RIA). I will also discuss these relationships in more detail in a future post.

Regular readers (if any should chance to return after my long absence!) will know that I will be focusing on service level management, and that I will be situating those service level issues within a wider context of application usability.

This is a broad topic, yet (as far as I can tell) a little explored one. I have been researching this subject area for almost 3 months now, and although you can find literally thousands of opinions and analyses of Ajax, RIAs, and Web 2.0 on the Web, very little of it deals with questions of performance management or SLM. So I'm not sure how many posts it will take before I run out of interesting things to write about. I suppose this may be considered to be an advantage of specializing in a field that most people take for granted. However, if you are planning to implement a Rich Internet Application, it would be a big mistake to take its performance for granted.

That was the theme of an introductory article I wrote recently for eCommerce Times, entitled Web Applications: Richer or Poorer? That article is a good place to start; it will give you a brief overview of the subject matter I plan to cover here in later posts, and introduce my world view. A key conclusion was that companies that decide to employ the technology must be prepared to invest more in design, development, testing and management to make it successful. In future posts I will justify and elaborate on that statement.

Climbing the SLM Maturity Ladder

2005-11-16T08:01:00.000Z

Last week, one of my posts introduced Peter Armstrong's paper about IT-Business alignment. BMC has just published a new whitepaper by Peter -- Taking IT to the Next Level -- about the challenges IT organizations face as they evolve from a cost center to a creator of business value.

Peter uses the Gartner IT Management Process Maturity Model, pictured above, to represent the five possible maturity levels of IT organizations. This is an excellent way for companies to measure their progress in implementing systematic Service Level Management (SLM) processes. In his whitepaper, Peter is focusing on climbing the ladder from Level 3 (Proactive) to Levels 4 (Service) and 5 (Value).

It occurs to me that a Maturity Model may be the corporate equivalent of a Twelve-Step Program -- to get better you must first admit you have a problem. That's why so many companies are still stuck at Level 1 (Chaotic), while others who are forever coping with the problem of the day at Level 2 (Reactive) would be really happy to establish to firm foothold at Level 3!

Web Usability Books [4]

2005-11-15T09:00:00.000Z

Today I am continuing my review of Web Usability books, from the perspective (described here) of someone who believes that Performance Matters:

4. Usability for the Web by Tom Brinck, Darren Gergle, and Scott D. Wood

Subtitled Designing Web Sites That Work, this book is about managing the design process, with the term design being used in its widest sense. In the introduction, the authors define Usability as the product of several design goals: functionally correct, efficient to use, easy to learn, easy to remember, error tolerant, and subjectively pleasing.

I particularly like the emphasis on the process of producing and maintaining a Web site. While The Design of Sites contains an introductory chapter summarizing the steps of the development process, here those steps are used as the central organizing principle for the book. On his own site, lead author Tom Brinck explains why:

Darren, Scott, and I were all involved in the genesis of Diamond Bullet Design, a web design company founded in 1996 with the goal of bringing usability to the web, and together we worked out how a web design and development process should reflect usability at every stage, while making this a feasible process within the business constraints of actual development.

The book's introduction continues:

To ensure high usability on our own Web projects, we defined a development process that incorporates proven techniques from software and usability engineering, graphic design, project management, and other disciplines. The process had to be practical and lean. It had to allow us to work on multiple projects of varying sizes with fixed budgets. It had to help us keep track of the details that can kill usability and destroy profitability. This book is about that process.

Tom's site provides some more detail:

Usability for the Web provides a comprehensive approach to the web design process, with a devotion to the principles of usability. The book is intended for web professionals and students of web design, with a practical focus.

The book is organized into 6 sections, according to the activities of the design process: Requirements Analysis, Conceptual Design, Mockups and Prototypes, Production, Launch, and Evaluation. We view Evaluation as an ongoing activity throughout the process, which is integrated at every step. Detailed chapters cover topics such as defining the target audience and target platform, user needs analysis, task analysis, information architecture, visual design, writing, software development incorporating usability, quality assurance, user testing, and usability inspection.

Numerous forms and checklists are included that are straightforward to apply in real-world design situations. These forms are designed to create a reliable, complete, and repeatable design methodology.

The publisher, Morgan Kaufmann, lists the target readership for this book as Web site designers and developers, Web site project managers, usability specialists and information architects, user interface designers, and graphic designers. While any of these readers may want yet more detail about their specialty, this book does provide almost 500 pages of carefully selected, well organized, well written, and attractively presented material. Indeed, the book itself is a fine testament to the integrity of Tom Brinck's stated Usability Philosophy.

Finally, for those interested in Performance Matters, the authors devote a full chapter to Usability in Software Development. This includes a statement about the crucial importance of Response Time, a short discussion of what contributes to it, and a sidebar on Why Software Engineers are Critical for Usability.

Granted, one 24-page chapter cannot cover the subject of performance properly. But this is the only book on usability I have ever seen that explains 3-Tiered Architectures, and how to map the flow of control and data through those tiers using static design diagrams and dynamic request traces. And its discussion of latency diagrams highlights the need for a systematic way of analyzing where the time goes, like the one I wrote about in an earlier post on The Web Site Response Time Model.

After reading the story of Tom Brinck's diverse background (Apple, Stanford MS in Computer Science, Toshiba, Bellcore, Michigan MA in Psychology, founder of Diamond Bullet Design), it is easy to understand why his ideas about usability and his book are so balanced and well rounded.

Thoroughly recommended!

[Next book review: Speed Up Your Site ...]

Web Usability Books [3]

2005-11-15T06:00:00.000Z

Today I am continuing my review of Web Usability books, from the perspective (described here) of someone who believes that Performance Matters:

3. The Design of Sites by Douglas K. Van Duyne, James A. Landay and Jason I. Hong

As its title suggests, this book is written for anyone involved in the design of a Web site. In the preface and on their site the authors say: Its focus is tilted more toward Web design professionals, such as interaction designers, usability engineers, information architects, and visual designers. Since the book comprises over 800 pages of densely-packed design advice, its main audience is pretty clear.

But the authors claim that it is a resource for anyone on a Web development team, from business executives to advertising managers to software developers to content editors. Its value to this wider audience is well explained by Mike Tarrani in his "spotight" review of the book on Amazon, so I will intersperse Mike's comments below:

Tarrani ... Nearly every book on user interface and site design I've read is aimed at the professional designer who understands the nuances of color, fonts and graphics elements, as well as aesthetics in general. Many of the subtle points are lost on the non professional.

The first 100 pages (Part I) are devoted to Foundations of Web Site Design, and include chapters on Knowing Your Customers, Involving Customers with Iterative Design, and the Site Development Process. The authors describe the last of these as ... a rough guide to designing, implementing, and maintaining a Web site. I concur -- it is a useful summary of the major stages, but no more than that. The real meat of the book is Part II, which comprises 517 pages. Here, 91 Design Patterns are organized into 12 pattern groups (A-L), and illustrated with examples of real Web sites.

Tarrani ... It leads you through each example, showing you how a particular design or design concept works and why. This is akin to the Rosetta Stone for the non-professional designer because the authors do not assume any talent of skills in design, and subtle points are highlighted and clearly explained. Because of this approach I finally understood concepts that had eluded me in the past.

In addition to the clear explanations that distill design into patterns, the book is lavishly illustrated, using copious full color examples and a structured format that gives the background, frames the problem and provides a solution to each of the 12 design goals.

Not everyone agrees, of course. If you study the Amazon reviews, you'll find a few that rate the patterns as shallow, incomplete, or out-dated. But that is inevitable given the authors' approach -- a book like this can never be complete or up-to-date, because the technology and people's use of it is a continually moving target. In fact, the authors state this in their introduction to Part II. In my view, they made a valiant attempt to classify important Web design patterns, and the great majority of reviewers do find a lot of value in the result.

Tarrani ... Material in the appendices is also invaluable, including advice on running usability evaluation, and associated plan outlines and forms. For a development group this is an extra bonus that will make it easier to incorporate the principles in this book into a quality process that gives customer-focused usability the same weight as technical quality criteria.

I'm so enthusiastic about this book that I've recommended to the company for which I work that a copy of this book be provided to each of our developers who are programming wizards, but who stumble when it comes to the user interface.

My own special interest in Performance Matters is served by the last pattern group, Speeding Up Your Site, which comprises five patterns: Low Numbers of Files, Fast Downloading Images, Separate Tables, HTML Power, Reusable Images. While these are not original or surprising, they certainly do address some of the most common page design problems that can affect performance.

[Next book review: Usability For The Web ...]

Web Usability Books [2]

2005-11-11T08:01:00.000Z

Today I am continuing my review of Web Usability books, from the perspective (described here) of someone who believes that Performance Matters:

2. Designing Web Usability : The Practice of Simplicity by Jakob Nielsen

Jakob Nielsen is a famous usability guru. He writes, speaks, and consults on Web site design and usability, and his 1999 book is a classic. It is surely the best-seller on this subject, having been translated into 21 languages. According to the New Riders Press publisher's introduction over 250,000 Internet professionals around the world have turned to this landmark book. Nielsen's site lists plenty of good reviews, this one from BusinessWeek being just one example:

KEEP IT SIMPLE. Nielsen ... is a usability engineer and Web-design consultant given to strongly held ideas and sweeping statements, like fast response times are the most important criterion for Web pages. His certitude might be insufferable if he weren't right so much of the time.

The basic principles of good design, according to Nielsen, are very simple: If your pages don't load quickly, your customers won't wait. If customers can't find what they want, they won't buy it. If your pages are confusing or hard to read, customers will look elsewhere.

But while easy to state, such general principles are very hard to implement. Here Designing Web Usability shines, with hundreds of screen shots of actual Web pages, successful and unsuccessful, with a narrative that highlights just what their designers did right or wrong. You'll probably find plenty to disagree with in Nielsen's conclusions. But you whether you're a designer or someone with responsibility for your company's online presence, you'll come away from the book with a much deeper understanding of how to use the Web as an effective sales and communications tool.

Yet, as BusinessWeek hinted, Jakob and his book generate strong opinions both pro and con. The book's subtitle, The Practice of Simplicity, sums up his world view, and also accounts for much of the controversy. Jakob writes in his book's preface:

I wrote this book with two clear goals. The highest-level goal was to improve quality of life by reasserting humanity's mastery of technology. On a day-to-day basis, you may think of usability as a way to increase sales (for public websites) or productivity (for intranets), but in the aggregate, if we make websites and intranets easy to use, we will increase users' quality of life by eliminating a lot of frustration and the feeling of inadequacy that follows every time you are stumped by a computer.

My more immediate goal was to promote a new philosophy of web design: simplicity. When the book was originally published, this was a controversial goal and I was often in the distinct minority at industry conferences. Eyeballs ruled the day and it didn't matter much whether users could actually accomplish anything. I am now happy to declare victory. Usability has become accepted as an important component of almost all professional web design projects, partly as a result of the success of this book, but mainly because it works.

On the second page of Chapter 1, Jakob writes:

Art Versus Engineering: There are essentially two basic approaches to design: the artistic ideal of expressing yourself and the engineering ideal of solving a problem for a customer. This book is firmly on the side of engineering. While I acknowledge that there is a need for art, fun, and a general good time on the Web, I believe that the main goal of most Web projects should be to make it easy for customers to perform useful tasks.

As you can imagine, I am in violent agreement with this viewpoint! However, Jakob's approach to Usability comes in for plenty of criticism from his own profession. To see why, you need only browse Jakob's web site. People like me who use the Web primarily to find information, and who value responsiveness, don't mind the way it looks, as long we can find what we are looking for. But most designers and graphic artists, who value aesthetics, hate the rather primitive appearance of Jakob's site -- and as a result probably feel justified in ignoring his advice.

For examples of their views, scan the Amazon book reviews. A blog post, on How Usable is Jakob Nielsen? by usability professional Frank Spillers, which is referenced in Jakob's Wikipedia entry, provides a good summary of the anti-Nielsen viewpoint. And Spillers' company, Experience Dynamics, has a simple, responsive, and nicer-looking site, too!

Having said that, this is a book review and not an assessment of Jakob Nielsen's place in the pantheon of Web designers -- if there were an actual Pantheon for these modern-day gods, he would surely have a place.

Jakob makes no pretense of offering advice on how to construct a site, saying ... this is not a book about HTML or how to draw an icon or other Web implementation technology. But he does devote 9 pages of his chapter on Page Design to discussions of the importance of fast and predictable response times, quoting Robert B. Miller's classic 1968 research paper on Human-Computer Interaction, which I wrote about previously. So I recommend this book, not just because of its classic status, but for the emphasis Jakob places on the Responsiveness dimension of Usability.

[Next book review: The Design of Sites ...]

Web Usability Books [1]

2005-11-10T21:09:00.000Z

I have twice promised to recommend books on Web Usability, so it's about time I got on with it. I have organized the books I like on a scale from simplest to most comprehensive. Today I will review the first, and simplest, of my recommendations.

Before we start I have to point out that I am not an expert in the subjects these authors devote most of their pages to, which I have labeled Clarity in my four-dimension usability model. My particular interest in writing these reviews is not primarily in the authors' expertise in the Clarity dimension, but in the extent to which they also acknowledge or discuss the dimensions of Availability, Responsiveness, and Utility.

[To understand what I mean by that, you must first read at least yesterday's post on The Dimensions of Usability. The previous post, where I introduced this model, was Web Usability: A Simple Framework.]

That is not to say that I am completely ignorant about their subject matter, either. I own a copy of each book I will discuss. I did my homework before buying them. I have spent time studying them. So I don't have a problem in offering my observations about their content. It's just that I do not have to use a lot of this material on a day-to-day basis in the course of my job. So think of my reviews as Web Usability books for people who don't already specialize in it.

1. Don't Make Me Think, by Steve Krug

Subtitled A Common Sense Approach to Web Usability, this is an excellent introduction to the subject. It is widely praised for its simple approach -- in fact, it's hard to find a bad review of this book anywhere, starting with over 250 reviews on Amazon. At under 200 pages, the worst criticisms are from people who think it should be longer. Published by New Riders, the style and mix of text and illustrations is similar to the Visual QuickStart Guide Books from Peachpit Press. If you like those, you'll probably like this. The first edition that everyone likes was published in 2000, but you can see from Steve's site that he has a 2nd edition just out. I have ordered it but not seen it yet. Apparently it is updated, and is about 30 pages (and three chapters) longer. If you're buying a copy from Amazon anyway, you can double the author's royalty by ordering it though his site.

Finally, as you might have guessed from its title, this book is all about site Clarity. It doesn't have anything to say about Availability, Responsiveness, or Utility.

[Next book review: Designing Web Usability ...]

The Dimensions of Usability

2005-11-10T02:30:00.000Z

Whenever specialists attach technical meanings to everyday words, they tend to create confusion and make their profession harder to penetrate. Some professions are masters of this. If you heard the words party, service, and cases in a sentence you might be looking forward to a celebration -- unless they were spoken by your lawyer.

In the world of information systems, Usability is a case in point. Model is another -- especially since it can be a noun, a verb, or an adjective. All the same, today I am going to use a model to help me discuss the concept of usability!

In a previous post, I presented a framework for thinking about Web Site Usability. I divided usability into four dimensions, each of which is essential to the ultimate success of any site -- Availability, Responsiveness, Clarity, and Utility. The graphic illustrates the four dimensions of usability, and its layout reveals some further characteristics (Technical or Functional, Core or Refinement) of the four dimensions. This further classification scheme is not strictly essential to an appreciation of the underlying four-dimension model, but may be helpful when thinking about differences among the four aspects.

Horizontally, the Functional dimensions pertain to the purpose of a site and how it is used, the Technical ones concern its construction and delivery. Vertically, the dimensions I have labeled as Core represent the essential implementation of a site's function or of its delivery, the ones labeled as Refinement represent the form (or quality) of that implementation. For the architecturally minded reader, this distinction follows the classic saying of architect Louis Henri Sullivan that Form ever follows function. For the engineering-minded, the practical implication is this -- why worry about responsiveness if your site is not available consistently, and why work on refining the user interface if the site does not deliver the function the user needs?

Having said that, in practice people do recognize the need to address all four dimensions of usability, and tend to think about them about independently, and to work on them in parallel. This turns out to be both good and bad. It's good because it allows specialists in each of the four areas to focus on what they do best. Typically this means that:

Business experts provide the content or specify the behaviors that are the site's purpose (Utility).
Design and Usability specialists make it easy for customers to navigate the site (Clarity).
Site developers build the site in ways that determine download speed (Responsiveness).
IT staff manage the systems that keep the site up and running (Availability).

But this division of labor also has a drawback -- it encourages Web Usability professionals to focus almost exclusively on what I call clarity. In fact, they have expended far more energy on internal debates about issues such as design vs. usability than on incorporating a focus on the other dimensions.

In one sense, I am objecting to something that is simply a matter of semantics. The English language meaning of usable (with a small 'u') is more inclusive than the professional definition favored by Usability experts (with a capital 'U'). As my opening paragraph implied, it may be confusing, but I can't expect an entire profession to redefine itself just because I don't like their choice of terminology! But given this difference in scope, it is interesting to see that the International Organization for Standardization (ISO) actually does define usability in a way that seems (on the face of it) to be closer to the viewpoint of a business:

ISO 9241-11: Guidance on Usability (1998)

Usability is the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.

This standard (which is part of the ISO 9241 series) provides the definition of usability that is used in subsequent related ergonomic standards.

ISO 9241-11 explains how to identify the information that it is necessary to take into account when specifying or evaluating usability in terms of measures of user performance and satisfaction. Guidance is given on how to describe the context of use of the product and the measures of usability in an explicit way. It includes an explanation of how the usability of a product can be specified and evaluated as part of a quality system, for example one that conforms to ISO 9001.

It also explains how measures of user performance and satisfaction can be used to measure how any component of a work system affects the quality of the whole work system in use.

Ultimately, what really matters to the business owner is their site's utility -- the degree to which customers can achieve specified goals with effectiveness, efficiency and satisfaction. And unless you address all four dimensions of usability, you will not accomplish that. Rather you will drive away all but your most loyal customers, and those who are forced to use your site for other business reasons. Those who have choices will not be satisfied. They will abandon your site and find other ways to get the job done.

At least where Web sites are concerned, the Usability profession has defined its role more narrowly. Most articles and books on Web Usability focus on the (many) aspects of clarity, and make only passing references to the importance of the other three dimensions. In my opinion, if writers and teachers would adopt the rather broader scope of the ISO definition (and that of the English dictionary), it would help to promote a more holistic Web site development process, and lessen the chance of the other dimensions being downplayed.

Some authors have done better at this than others. Tomorrow I will review a few books by leading authors on Web Usability, in the light of the four-dimensional Usability Model.

Armstrong on IT-Business Alignment

2005-11-08T08:01:00.000Z

If you research the business case for SLM on the Web, eventually you're sure find yourself reading something published by NextSLM.org, a site sponsored by BMC Software. One contributor is a former colleague of mine, Peter Armstrong.

Peter and I started our careers at IBM UK in the 70's as SE's supporting IMS customers. Although information technology has evolved a bit since then, our interests apparently have not diverged much. Peter now writes a blog called Adopting a Service (Management) Mentality, which focuses on the increasingly important domain of how business and information technology need to work together -- the area he is responsible for at BMC.

Buried in the archives of the NextSLM site is an essay by Peter entitled Why IT and Business Need to Talk: Two Nations Separated by a Common Language? I will extract some key paragraphs, adding my own emphasis in each section. Written (as best as I can tell) in 2003, it is peppered with stories of IT and business not communicating, for example:

Your IT department has spent days gathering all the information on server availability, and come to the board meeting ready to prove that they have been delivering 99.99% availability for the last week and cannot understand why anybody is complaining. Unfortunately the application is being used by online options traders who need a response time of less than 12 seconds in which to make a trade. Availability is meaningless to them without performance.

Peter sets out to explain why IT and business have to learn a common language. He writes about the cultural and historical reasons for the divide between the two sides, and the challenges they face in achieving the much-desired state of alignment. Being familiar with how IT works, he observes that:

Many IT departments focus on the technology and delivery of availability of platforms, databases and applications. But although all of these are important, it is how these elements interact to provide a business service that is the key issue. It is vital that the IT department understands not only the technology but also the way that the technology interacts to deliver service ...

IT needs to understand that its sole function in life is to enable the business to run better. This means that it is either helping to reduce costs, and/or it is helping you increase revenues ...
However, the IT department is between a rock and a hard place as they are being told to reduce costs, and by far the most important factor is actually quality of service. But when budgets are restricted, it is also the one factor that often gets pushed down the list of selection criteria. Some managers see low cost and high quality of service as being mutually exclusive but this need not to be the case.

This article claims to be the first of two parts, but its link to the second part -- How Good is My Service -- is broken. But Google found me this PDF, which seems to contain the original essay before it was split into parts. In the second half, Peter discusses several aspects of SLM. Looking first at Web sites, Peter highlights how vital it is that companies doing business online understand their customers' point of view. Someone needs to ask:

Who is using your IT systems? Why are they using them? What are you trying to sell them?
Is it easy to contact you if they have problems? Is the data up-to-date?

Are the systems available (and this includes performance)?

What are the availability and performance requirements (some systems perform better than required, which is a waste of money)?

Do you have Service Level Agreements (SLAs) in place and are you measuring against them?

Are you measuring from the end-user point of view or from your own point of view?

Turning his attention to service management and the nature of SLA's, Peter writes:

The cornerstone of any managed services contract is the service level agreement. This is meant to be the yardstick that measures the performance of the service provider, but unless the agreement is written and reported on with measurable and meaningful values it can lead to very difficult situations ...

I was visiting a major customer last year, whose IT operations are outsourced to one of the major outsourcing companies. The contract is, of course, couched in terms of CPU (processor resources) used, number of housekeeping jobs run, number of tape mounts etc. -- totally and utterly wrong in my opinion. The bank wants a service -- how the outsourcer actually implements it technically should be of no concern as long as it is safe, reliable and conforms to the business SLAs ...

Reporting is another critical area that can get overlooked. A good IT department will not only provide good service but also prove it by providing meaningful reports. These reports should relate directly to the service being provided to the customer and also show the value that the IT department is bringing to the equation. They should be couched in language that the business people can understand -- Oracle availability is meaningless, ability to print invoices on time is significant.

Like me, Peter is a strong advocate of measurement tools, which should be deployed within a systematic SLM framework such as ITIL. [Since we work for companies that sell tools and services, you might be inclined to discount our views. But I'm sure Peter would agree with me that this is a "chicken and egg" situation -- we work for tool vendors because we believe in the value of their tools and services, and not vice versa.] Peter writes:

At a technical level the key issue is how the IT department actually proposes to manage the environment. The use of industry-leading solutions and best-practice methodologies should be common to all good service providers, but they are surprisingly often absent.

Some IT departments see tools merely as a way of reducing their cost of operation by reducing the number of people required to manage the environment. However, the use of tools to manage technology can greatly increase the availability and performance of the infrastructure. By exploiting the functionality within the toolset and applying that functionality to best-practice standards such as ITIL, the business -- and ultimately the customer -- receives the benefits.

Through implementation of sensible IT processes, combined with business-related tools and methodologies, the advantage to the business is the true and correct exploitation of the IT investment. This is why the two parties need to learn to talk to one another.

It is clear that Peter's thinking has subsequently found its way into BMC's more sanitized (and glossier) marketing materials on this topic, like his whitepaper on Seven Strategies for Enabling IT to Activate the Business, and this corporate strategy brochure on Business Service Management. But Peter's original essay reveals better the strength of his personal belief in the importance of SLM to the business.

I share that belief, and endorse his conclusions.

SLM: Learning from Dot-coms

2005-11-07T08:01:00.000Z

In life, wisdom is generally thought to come with age. But in business, years of experience do not always lead to best practices. This often seems to be the case when it comes to the systematic application of performance management or service level management (SLM) practices.

Lately I've been working on the business value of SLM. In particular, I've been focusing on how to establish the value of SLM activities devoted to improving the availability and responsiveness of Web sites and e-business applications. In the process I've been speaking to several companies about how they approach this question.

Personally, as I said in my previous post, I'm convinced that there is always a good business case for SLM, if it's implemented properly. But I've spent many years advancing this proposition, so my opinion probably does not count for much. For any advocate, what matters more is the listener's own predisposition. In this case, regardless of how strongly a company seems to be embracing e-business approaches, their history and culture can still affect their readiness to embrace and justify SLM programs to support that business.

It's now more than five years since the dot-com bubble burst in 2000, wiping out billions of dollars that had been invested in speculative Web businesses. The survivors may have picked a better idea for an online business, but more importantly, they knew how to make a profit running that business. Those who did not quickly joined the ranks of eToys, Pets.com, Webvan, and hundreds of other long-forgotten companies.

At successful e-business companies, because business is conducted entirely over the Web, the value chain connecting SLM process to site usability to customer satisfaction to revenue and profits is an ever-present fact of life. At older companies, where e-business components are now being added to existing offline businesses, the corporate culture is not nearly as attuned to that e-business value chain.

Years of investment in IT infrastructure and applications were designed primarily to deliver the back-office systems that supported "bricks and mortar" businesses. Back-office systems are used by company employees, not customers. Although poorly performing applications may annoy employees, and even lower their productivity, it's usually safe to assume that they do not drive away business.

But when the company starts adopting e-business, those old assumptions must change. Older companies must learn what successful dot-coms must already know -- SLM is important, because it affects the bottom line.

Performance matters!

The Business Case for SLM

2005-11-04T08:15:00.000Z

These days everyone has an opinion about aligning IT with the business. If you think I'm exaggerating, just Google that phrase and read a few of the 800+ references you get back. You will quickly see that this subject generates very strong feelings, the trajectory of which depends largely on whether the writer views the world through the prism of IT or the business.

Bob Evans summed up the strength of these feelings very well in the first sentence of his amusing Information Week column, A Trip To The Cafeteria--With The CFO! Contentious issues abound -- squeezing more productivity out of already overworked IT employees, cutting technology spending, the pros and cons of outsourcing, getting the business to understand what it takes to keep up with technological advances, not to mention Sarbox regulations. Or sometimes, just getting the business to settle on a direction, any direction -- N? E? NE? NNE? -- so that IT alignment can begin!

To express opinions hastily on this subject would be like chasing butterflies into a minefield. Fortunately, I can sidestep that minefield. The issue at the core of the debate is the value obtained by a business in return for its investment in IT. And no matter what your opinion about the politics of IT/business alignment, investing in Service Level Management (SLM) is always worthwhile.

SLM is not a passing IT fad to be adopted then discarded like the latest technical buzzword. It is an essential way to maintain and improve the quality and effectiveness of your business. And when your business is online, implementing a systematic SLM program can -- as suggested in the graphic above -- benefit both revenues and costs.

This is not a recent revelation, triggered by the debate about IT and business alignment. In 1998 I wrote an article on Controlling Application Performance: What Every CIO Should Know, which began ...

CIOs face complicated issues of data center infrastructure, a portfolio of widely differing application workloads, and rapidly changing business requirements. Strategically, CIOs need to react quickly to competition. Tactically, they must create common processes across business units to make collection and dissemination of information simpler and to slash the cost of new applications.

Compared with these concerns, application performance lacks glamour. Nevertheless, a systematic program of performance monitoring and tuning almost always produces surprising cost savings for large companies.

CIO sponsorship is vital for such a program to succeed. In the first place, only the CIO can negotiate the transfer of funds between the capital equipment, salary, and software budgets. Second, management prioritization and employee motivation are required to convince lower-level managers and technical professionals of the importance of application performance. The CIO can provide the necessary emphasis.

That article emphasized the cost savings achievable through application tuning, and little has changed since 1998 in that respect. In today's world of e-business, equally important opportunities also exist for system and application tuning efforts to increase revenues, by making an online business more available and more responsive to customers.

CIO sponsorship is still an essential first step to success with SLM, for the same reasons as before. As a CIO seeking to better align IT with the business (whatever that means this week), a proposal to beef up your SLM processes may seem at first to be just another expense to be placed under the cost-cutting microscope. Proponents of SLM programs should welcome such scrutiny. When the applications to be managed are important to the business, opportunities always exist for significant cost savings, revenue increases, or both. I would be willing to bet that the initial investment will more than pay off over time, in applications that help the business by serving customers better.

In fact, I'm convinced that there is always a good business case for SLM, if it's implemented properly. Future posts will consider some of the areas where SLM pays off, and ways of calculating the ROI of SLM investments.

Keeping a Public (Site) Healthy

2005-11-03T08:01:00.000Z

Browsing the Web can lead to useful insights, even when you don't find what you're looking for. Today I was searching for information about an aspect of Web site performance management. As often happens, Google returned many links to pages where the term performance management was used in contexts having nothing to do with managing Web applications.

On one site, Turning Point, the graphic shown here immediately caught my eye. It was so obviously relevant to my world of application performance management that I studied it for a while before noticing that Turning Point ... is an initiative of The Robert Wood Johnson Foundation and the W.K. Kellogg Foundation. Its mission is to transform and strengthen the public health system in the United States by making it more community-based and collaborative.

This sounds like a worthwhile cause, but in a profession whose technical, management, and political challenges I know little about. Nevertheless, the site's definition of performance management, and its description of the four key components illustrated in the diagram, written for the world of public health, apply equally well to the task of managing your Web site health!

To demonstrate the parallels, the following paragraphs are quoted directly from the Turning Point site, except that I have made some minor edits:

Performance management is the practice of actively using performance data to improve ~~the public's~~ your site's health. This practice involves strategic use of performance measures and standards to establish performance targets and goals, to prioritize and allocate resources, to inform managers about needed adjustments or changes in policy or program directions to meet goals, to frame reports on the success in meeting performance goals, and to improve the quality of ~~public health~~ SLM practice.

Performance Management components include:
Performance Standards: establishment of organizational or system performance standards, targets and goals and relevant indicators to improve ~~public health~~ SLM practice.

Performance Measures: application and use of performance indicators and measures.

Reporting of Progress: documentation and reporting of progress in meeting standards and targets and sharing of such information through feedback.
Quality Improvement: establishment of a program or process to manage change and achieve quality improvement in ~~public health~~ SLM policies, programs or infrastructure based on performance standards, measurements and reports.
A Performance Management System is the continuous use of all the above practices so that they are integrated into the organization's core operations. Performance management can be carried out at multiple levels, including the ~~program~~ application, ~~organization~~ Web site, ~~community~~ line of business, and ~~state~~ corporate levels.

It seems that whatever your professional focus, you can always find parallels in other fields. Those technical details you've been wrestling with to speed up your Web site may not be a hot topic of conversation at your next dinner party. On the other hand, the process of performance improvement seems to be a concept everyone can relate to.

If you chose your words really carefully, people may not even realize you manage a Web site. Just tell them you're a health specialist -- maybe the diagram above will help with your explanation. It worked on me!

Keeping it on the Road

2005-11-02T08:30:00.000Z

Yesterday I introduced the subjects of time series and probability. Today I want to show how they apply to one aspect of service level management -- working with service availability data, which you are probably collecting from several different measurement tools.

Suppose you need to improve the availability of a critical e-business application. And let's assume you have collected historical availability statistics, or estimates you are comfortable with using, for all components of your Web environment. You'll need these for the devices and services you control yourself, and for those you don't, like your ISP and its connection to the Internet. Hopefully, many of those numbers will be close to 100%. But if you are having service quality problems, at least some of those availability scores must not be good enough.

Barring catastrophic outages that bring down an entire data center, the components that support your Web applications generally operate independently of one another. That is, Web server crashes are unrelated to database server problems, which are not affected by a back-end application service being down. So if you know the availability of each one separately, it's intuitively clear that you should be able to derive an estimate for the availability of the whole interconnected system, for a given application. In other words, you can estimate of the level of application availability a customer experiences.

Here's how you can use the Web Site Availability Model to do that:

Based on my conceptual diagram, create a spreadsheet model listing all your services, where each row of your sheet corresponds to a service that's used to deliver your Web site or Web-based applications.

Check that the target application does not depend on any unlisted services. This would be most likely to happen in the bottom row, called Application Services, because what goes in that row will vary from one application to another.

For each separate page of the target application, set aside one column of an availability model.

In each column, highlight only the cells that correspond to services needed to serve that page. These cells are the only ones you will use in this instance of the model. Not every service listed is required for the delivery of every page. Simple applications may require only a small subset of the services you manage.

In each row of your completed model, record the percentage of down time (100% minus availability) of each service in the first cell where that service is used (highlighted). At this point, the spreadsheet is already a useful document that can help refine your focus and tackle the biggest problems.

Summing the percentages in each column will now give you the probability that your application will fail on any particular page. A customer may be equally upset if your home page is down, the product catalog is down, or the checkout process times out. But to improve overall application availability, you need to know which pages face the most problems.

Summing the column totals for whole model will tell you the overall chance of an application failure.

How's the Weather in Your Data Center?

Although it may not be immediately obvious, the above method assumes that your devices and services behave a lot like the weather in California (to use the example I described yesterday). Just substitute up and down for sunshine and rain! Service outages do not happen for a single instant, then go away. Normally a service is 100% available, but when it goes down (becoming 0% available), it is stays down for some time.

Therefore, the proposed method assumes that if a service is available the first time it's needed by any page of an application, it will remain available for all subsequent pages. Making this assumption greatly simplifies the challenge of aggregating device and service availability data from disparate sources.

If on the other hand you have reason to believe that an underlying service behaves independently from page to page of a Web transaction, the weather analogy breaks down. In that case, you have a bit more work to do to complete the cells of your availability model. First you must count the number of pages of the transaction that use the service, then you must compute the probability that none of those pages will fail when using the service.

This situation is what statisticians call a sequence of independent trials, like successive rolls of a dice. For n pages, the probability of no failures is service availability raised to the power n. For example, if a service is 90% available overall, the probability that two pages will succeed in using that service is (.9)(.9) = .81, three pages (.9)(.9)(.9) = .729, and so on. Subtracting that result from 100% gives you the percentage of down time for that service, during your application. Now proceed as in steps 5-7 above -- enter this result in the first highlighted cell (the first page requiring the service), sum the component unavailability data by page, and sum the columns to derive the overall probability of an application outage.

Note that all the possibilities for a service failure have once again been recorded in the first column (page) that uses the service, which does not matter if you are focused only on overall application availability. If you really want to estimate the probability of a particular page failing, you will have to spread the probability of failure across all the pages that use the service. This requires computing the conditional probabilities of each page failing, given that its predecessors did not. That, as they say in the textbooks, is left as an exercise for the reader.

Details, details, ...

Astute readers may have noted that my suggested method glosses over a couple of important details:

If you use a group of parallel servers (for example, with load balancing), you should treat the group as a single logical service in this model. The customer does not care whether one of those servers is up or down, if they can still obtain service from another server in the group. Of course, that is one reason why you are using parallel servers -- the goal is to achieve 100% availability for this service, regardless of whether an individual server is up or down.

In practice, customers may take different paths through your applications, where those paths (usage scenarios or use cases) require different sets of devices and services. If service availability differs significantly based on application usage, you should treat each scenario a separate application, or add weights to your model. Then you can combine the different use cases and come up with an overall availability estimate for the application. How you deal with this kind of detail will depend on your purpose in using the model.

Finally, my aim has not been to provide a rigorous exposition of data center forecasting as an application of probability theory. And I am aware that a probability is a quantity between 0 and 1, whereas percentages run from 0 to 100%. But for descriptive clarity, it seems easier and more natural to use the terminology interchangeably. But if you are actually doing the calculations, be sure to pick one or the other and not to mix them!

Probability: The Rain in Spain ...

2005-11-01T09:00:00.000Z

Today I was going to write about probability theory, statistical independence, and correlation, and their importance when analyzing performance data. But I know those subjects have a tendency to induce severe attacks of fear and loathing. So instead I'm going to talk about the weather -- specifically about the difference between British and Californian weather.

When I came to the US, having previously experienced only the notorious changeability of British weather, I was quite surprised to discover that Californians enjoyed almost the exact opposite.

You may have heard the song It Never Rains in Southern California (but man it pours). And if it's sunny today, chances are good it will be sunny again tomorrow -- and probably for the rest of the week too! But during the winter months in Northern California it often rains for several days. And when it's raining, expect the rain to continue for a few days.

A friend of mine used to joke that the talents of weather forecasters were wasted here. British weather forecasters are equally redundant, but for the opposite reason; it's almost impossible for them to get it right. To cope with this, they have evolved umpteen ways to say unpredictable -- sunshine with occasional showers, cloudy with sunny intervals, overcast but brightening later, and so on.

Maybe you're wondering why I am telling you this, since you probably already know why Surfin' USA is a lot better known than Surfin' UK. Or maybe you've figured out that my sudden interest in the weather is really just a cover for my original purpose of discussing probability theory. So let me explain.

Consider any weather-related variable that's measured daily, like hours of sunshine, or inches of rain. The technical term for this set of measurements is a time series. Suppose you want to analyze such a time series and use it as a basis for forecasting tomorrow's weather (and assume for purposes of this example that no other meteorological data exists). If the two conditions you care about are rain (R) and sun (S), then you can do a lot better than simply calculating the long-term probability of rain, which a statistician writes as p(R).

That naive approach might be OK for forecasting the weather in Britain. But not in California, where a forecaster needs to consider conditional probabilities -- the probability of an outcome given what we already know, which is written p(outcome|condition). If it's raining in California today, what really matters are the conditional probabilities of rain tomorrow [p(R|R)] and sun tomorrow [p(S|R)].

For a time series, these kinds of conditional probabilities depend upon a statistical property of the series called independence, and its opposite, autocorrelation. When the results of successive measurements are independent, knowing the history does not help you to forecast future outcomes. This is the challenge facing the BBC's weather forecaster, whose unpredictable weather is such that p(R|R) = p(R|S) = p(R).

But for the weather forecaster in Sacramento, California, prior knowledge is crucial. During a typical year it rains on 58 days, so p(R), the probability of rain on any given day, is 58/365 or 0.158. However, those 58 days are not randomly spread across the year. Like the Aspens in Colorado, they come in clusters. So if it's raining today, the probability of rain tomorrow [p(R|R)] is significantly greater than 0.158, because for Sacramento weather, the time series is autocorrelated.

So ... this has been today's science lesson from your Web Weather station. Tune in again tomorrow, and I will explain how understanding the statistics of weather forecasting can also help you forecast your Web site availability. And you thought those weather forecasters on the nightly news were just there for comic relief!

The Value of Reference Models

2005-10-31T22:00:00.000Z

A reference model establishes a shared foundation -- a frame of reference, or conceptual framework -- that can then be used to structure subsequent discussions of a subject. For more details, the best resource I could locate online is this discussion of the OASIS SOA Reference Model. Its focus is different from ours, but it contains some good general points, including:

A reference model is an abstract framework for understanding significant relationships among the entities of some environment.

A reference model is based on a small number of unifying concepts and may be used as a basis for education and explaining standards to a non-specialist.

A reference model is not directly tied to any standards, technologies or other concrete implementation details, but it does seek to provide a common semantics that can be used unambiguously across and between different implementations.

In my two previous posts, I presented reference models for Web application availability and response time. These models enumerate the major components that determine the performance of any e-business application. This emphasis on identifying the contributing factors reveals another essential characteristic of all reference models -- the need for a model to be complete. Reference models for Web applications will be of limited use unless they offer a way to identify all the components the affect the availability or responsiveness of any Web-based application.

Apart from their value as teaching tools, I see many potential uses for these reference models, no matter what aspect of SLM you need to focus on. By showing how the performance of the whole application is determined by the performance of its component parts, they can suggest:

ways to determine the design goals and service level objectives (SLOs) for components

the degree to which competing design ideas might improve performance

frameworks for designing a performance monitoring program

methods for predicting or summarizing overall availability or response times

where to focus remediation efforts, identifying component(s) most in need of improvement

ways to systematically evaluate and compare competing sites

frameworks for comparing the likely impacts of different technology choices

For example, the CMG paper in which I first published the response-time reference model discusses examples of how e-commerce application response times can be improved in three ways: by reducing the overall number of components, by speeding up individual components, and by moving some components off the synchronous response time path.

Of course, a reference model will only take you so far. Because of their level, these models are designed only to break apart application availability and response time into their major components, not to reveal everything that goes on "under the covers." For that level of understanding, each component must be further dissected, as I did when discussing the components of Web response time in the CMG paper.

But a reference model is usually a very good place to start from, no matter what your particular focus.

Sharpening the Saw

2005-10-29T06:45:00.000Z

If you've read The Seven Habits Of Highly Effective People by Steven R. Covey, then you may remember that the seventh habit is Sharpening the Saw. The idea is that you should always allocate some time for personal renewal.

Because I find my work interesting, or make it so by treating new tasks as a challenge to be mastered, I have a tendency to forget about the seventh habit. But today a colleague reminded me about this, asking if my blog was fun or work related. Of course, my response was "it's both" -- and it is. But this exchange got me thinking that I did need to sharpen the saw from time to time. My own performance matters too.

I have therefore decided that from now on I will publish the blog only from Monday to Friday, and take the weekends off. Even if I do spend time writing over the weekend, I'll save the posts for publication during the week. It's not as if I'm going to be covering news that can't wait for a couple of days. So if you take time off to sharpen your own saws, you won't be missing anything.

Have a great weekend!

The Web Site Response Time Model

2005-10-29T02:00:00.000Z

Yesterday I introduced a reference model for Web site availability, and promised to write more about its uses today. But I have since decided to first introduce the equivalent model for Web site response-time. Then I will compare and contrast the uses of the two models, because -- although similar -- their application is not identical.

The response time reference model illustrated by the diagram is a slightly updated version of the one described in a paper (E-Commerce Response Time: A Reference Model) I presented at the CMG conference in 2000.

A quick comparison with yesterday's post will show that the two models are very similar in structure. (And in appearance, thanks to my use of cut and paste to create the diagrams!) The two reference models have a similar purpose, and function at a similar level -- concepts discussed on page 2 of the 2000 paper:

Purpose: Each model proposes a way to subdivide a single quantity (Web application availability or response time as experienced by a site user) into its underlying components. That subdivision is essential because -- to produce the overall result a user experiences -- each of those contributing components must be separately designed, built, monitored, and managed.

Level: The two models share a common horizontal dimension. As the 2000 paper says: In an e-commerce application, each business transaction is implemented by means of a sequence of Web pages. And their vertical dimensions serve the same purpose: To partition ... the process by which a single Web page is served.

Because of differences in the nature of availability and response time, the vertical dimensions of the two frameworks are different, and their cells have slightly different meanings and uses. Yesterday I explained that in the availability model, cells represent distinct components of the Web infrastructure that must all work together to deliver a Web-based application to a customer. In the response time model, the cells represent distinct stages that together comprise that delivery process.

So the first (and simplest) use of these models is simply to identify, for any Web-based application, which cells contribute to availability and response-time. The next use is to record or predict the magnitude of those contributions. This is where the differences between the two quantities become most apparent:

Response Time: Response time components are simple to work with. The sum of the cells in each column is the total time a user spends interacting with that page, and the overall sum for a completed matrix is the time it takes to complete a task using that application.

Availability: If we want to adopt a similar approach to the response-time framework, we see immediately that we must use the cells to record unavailability, because we want to aggregate each component's contribution to the overall time the site or application is down.

But if we were to record the percentage unavailability of a component in every column where it is used, we would still run into problems because we would be counting the same overall percentage more than once. For example, imagine a Web server that is down 25% of the time. If we were to record an outage contribution of 25% for four or more pages of a Web transaction, and add these contributions, we would obtain an overall unavailability score in excess of 100%, which is impossible.

How do we fix that? If instead we record the outage percentage (or time) for a component in only the first column in which that cell is used, we avoid the problem of double-counting. [There is a sound technical justification for this method, which I will describe in a future post]. And because availability percentages are usually close to 100% (outage times are relatively short), we can usually add up all the cells in a model to produce a reasonable and useful upper bound (worst case) estimate of overall unavailability.

Admittedly, the various components typically incur problems independently of one another, so that in practice some outages may overlap. Overlapping component outages would make the true overall outage time (or percentage) less than the worst case estimate we obtain by simply adding the cells. But as long as all component availability rates are high, the error will be a minor one that we can live with if it makes the availability framework easy to use.

Uses of these frameworks will be my next topic.

The Web Site Availability Model

2005-10-27T23:29:00.000Z

I am a mathematician at heart, and all mathematical conclusions are derived by deductive reasoning from a set of an initial set of assumptions (postulates or axioms). So it is natural for me to want to establish a clear foundation on which to build any discussion of performance matters.

To discuss matters of performance and SLM, we need to establish two kinds of frameworks: one for performance quantities (the things we must measure and manage), and another for the processes through which we conduct our measurement and management activities.

Up to now, I have been focusing largely on SLM process frameworks (like ITSO), and process details like how to select competitive response-time objectives for your site and how to report on them. Yesterday's framework dealt with how to use performance data, not with the data itself or how to collect it.

Today I am turning to the quantities that comprise performance. In my simple Web usability framework, I noted that customers have two basic needs -- Availability and Responsiveness. I am now going to expand upon those two aspects of performance, beginning today with a framework (or reference model) for thinking about site availability.

The model is illustrated by the diagram above. It is a matrix whose row and column labels are fairly self-explanatory, I think. Its cells represent common Web infrastructure components that must all work together to deliver a Web-based application to a customer. Not every page of every site will require every cell -- for example, displaying your site's home page may require only the top four cells in the first column. For more complex eBusiness applications, more of the cells will needed.

I developed this reference model in June 2005, while working on a presentation. I had previously created a similar one for response-time (E-Commerce Response Time: A Reference Model -- a topic for a forthcoming post), and I saw that my approach could also work for availability. At the time I could not find anything like it on the Web. Then last week I stumbled across an excellent paper, Application Availability:An Approach to Measurement by David M. Fishman, originally published in 2000. It includes a diagram (Figure 2. Service Decomposition) similar to the first column of mine, and the following discussion:

... a Web-based application can be decomposed into a set of measurement points for service-level indicators. A user or client of the application performing a transaction depends on all the lower-level layers to complete a transaction.
In this case, a user or client (i.e., a human or a browser) establishes a connection with a Web server over a network. The Web server connects with the application server, which processes business logic. The business logic in the application server connects to the DBMS for data retrieval as appropriate. And, of course, the DBMS runs on the operating system; it is only as available as the operating environment on which it executes.
Service availability can be measured or tracked only as a subset of the complete, end-to-end stack. With a design that allocates sufficient independence between layers, it's possible to speak of the availability of a series or set of services, each of which is a subset of the user's requirement to be up and running from end to end.

This is a very clear explanation of the ultimate point of an availability reference model: to deliver any service successfully to the customer, every cell that is needed must be available. So the reference model helps you understand better the diverse components that you must manage, to achieve your overall availability objectives. I will expand on this tomorrow.

The ABC's of Measurement Data

2005-10-27T04:30:00.000Z

In previous posts I have focused mainly on setting response time objectives as an aspect of site usability, and on the Service Level Management process. I turn now to some discussions of performance measurements, and how to get the maximum value from them.

In my experience, companies usually have lots of measurement tools. Granted, some of them do sit on the shelf unused, but many are in use -- some even collecting data continuously. Despite all this data gathering, the value obtained from the data is often a lot less than it might be.

Today I will describe a framework, illustrated on the left, for addressing this concern. This diagram does not represent a process, but rather a conceptual framework. It comprises seven potential ways of exploiting performance data, each one involving a greater level of sophistication than its predecessor. As a mnemonic, the names of the levels bear the initials A-G. (A stroke of luck, aided by careful selection of terminolgy).

Aggregate: The essential starting point. Raw data has its uses -- you need individual data points to investigate exceptions, and scatter plots do reveal some patterns. But most uses for performance data demand summary statistics. Availability percentages, and the Apdex metric (a new way to report response times) are just two examples of many. All performance analysis tools, except those specifically intended only for detailed tracing, support this level.

Broadcast: Once your tools have summarized the data, it does absolutely no good unless it is sent to someone who looks at it. It's amazing how many so-called performance monitoring systems break down here -- data is collected, summarized, then filed away, never to be looked at. In the 70's it could be found in a pile of computer printouts collecting dust behind someone's desk, today it's in a file somewhere on the network. That does no good. Stewart Brand coined the famous phrase information wants to be free. My corollary is information about performance wants to be seen! It needs to come out of the server closet and be put on display in a dashboard, where it can be useful.

Chart: If a picture is worth a thousand words, then a chart is worth a thousand spreadsheet cells. A well-chosen set of charts and graphs can help you see the patterns in your aggregate statistics, turning your data into information. Spotting a pattern in a table of numbers is as difficult as spotting Waldo in a maze of comic-book kids.

Diagnose: Patterns (whether found manually or by a tool) are the key to detecting problems (alerting), isolating in which major component of your application or infrastructure those problems lie (triage), and finally discovering (diagnosing) their causes. Over time, tuning your charts can get you through this process faster. In practice, there are very few truly new problems, but we have a tendency to fix and forget, so they come back to bite us. Hold post-mortems, and refine your process.

Evaluate: In polite society, maybe comparisons are odious -- or odorous, depending upon who you are quoting. But in the world of SLM, comparisons, far from being undesirable, are essential. Reasonable objectives will be defined by the prevailing Web environment and your competition, and the essence of performance management is to continually evaluate your site against those objectives. Ultimately, user comparisons will help determine your site's success or failure.

Forecast: Projecting future performance is vital component of SLM, otherwise you will be forever dealing with surprises and crises. And understanding your systems' and applications' performance today is the only starting point for projecting how they will behave under load tomorrow. Or during the holiday season. Or when marketing launches that big product promotion. This is an aspect of SLM that my colleague and team member Donald Foss will surely be writing about here, when he gets a spare moment. (With the holidays on the horizon, he's busy these days helping his customers predict their future performance).

Guide: Many details must come together to create the ideal world of ITSO, in which you consistently meet IT service levels while minimizing infrastructure costs and mitigating risks. Not the least of these will be the catalog of best practices you develop for yourself, in the course of measuring and understanding what makes your Web applications tick. These should be captured, documented, and institutionalized in your development processes, so that the hard-earned performance lessons of the past become guidance for the future. This is how the very best companies get the top -- and stay there. Look for my colleague and team member Ben Rushlo to share some of his experiences in this area soon.

I hope this taxonomy and short overview of the uses of performance data is useful. And if you do not learn anything new about performance, at least maybe some of the incidental references will be interesting. Finally, if you really think my scheme needs more letters, please write and tell me what they are. I may be an old dog, but now that I'm getting the hang of blogging, I'm ready to learn some new tricks.

Performance: Déjà Vu All Over Again

2005-10-26T03:00:00.000Z

A few days ago, an email from TeamQuest offered me a whitepaper on The Renaissance of Performance & Capacity Management in the 21st Century. The email points out that anyone involved in IT for a reasonable amount of time understands that the IT industry is cyclical and says that a data center renaissance is coming full circle.

This got my attention, and not only because it is redundant (coming full circle implies rebirth, the meaning of renaissance). But we've learned to live with worse, especially in direct mail. Anyway, what makes this particular life-cycle so interesting to me is that I have actually participated in it at every stage. Here is my "cliff notes" version of TeamQuest's history of Capacity Planning:

Early 70s: Mainframes were almost the entire IT industry. The hardware manufacturers generated reliable income streams by persuading IT departments to upgrade the entire machine when the CPU hit 70% utilisation. An IT manager typically did not have the tools, the credibility, or the courage to dispute a vendor's recommendation.

Late 70s: Performance and capacity management products allowed IT departments to better understand their mainframe systems. Companies also started relating IT to the business, by subdividing performance metrics into workloads based around applications.

The 80s: The new distributed era forced companies to install performance software just to keep up with new technology.

Early 90s: Capacity management provided the means for testing machine consolidation and disaster recovery scenarios as well as its traditional predictive “what-if?” capabilities. Companies created capacity management departments, software was in abundance and it seemed that performance and capacity management had finally come of age.

Mid 90s onwards: The Web gave birth to eCommerce, and IT became synonymous with the business. Companies now cared more about IT performance than ever before. Hardware was cheap, so money and hardware were thrown at performance problems in a frantic bid to make them go away. Not surprisingly, this rarely worked, and as a result, IT departments have been losing credibility.

Today: We have come full circle. An over abundance of hardware and a general lack of understanding as to how it affects performance means that performance and capacity management is now more important than it has ever been.

This is a pretty good summary of what I have seen happen over the course of my career as a performance specialist. So what's the whitepaper's punch line? It comes in two parts, actually. The first is the need for companies to adopt the approach of IT Service Optimization (ITSO), the five-step TeamQuest process shown in the chart. The goal of ITSO is to consistently meet IT service levels while minimizing infrastructure costs and mitigating risks.

While ITSO is a proprietary label, the principles TeamQuest is promoting are aligned with the ITIL industry standard and the ITSMF organization. These initiatives are designed to address the issues I described in my earlier post on SLM. This entire subject is a familiar one -- it occupies three chapters and 115 pages in my book, and I will be writing about it again in future posts (which will include reference to the second half of TeamQuest's conclusions).

For old hands in this industry, the idea of things 'coming full circle' is always interesting. Not always to be welcomed, but often so. Why? Because rebirth confers instant expertise -- we can reuse our hard-bought experience. The imagination races at the prospect! We will be asked to tackle problems we have already solved, to figure out the answers to questions when we have already seen the exam paper.

Or, foes who defeated us the last time are returning for another round, and this time we will out-fox them. As in Bill Murray's Groundhog Day, if you re-live the worst day of your life enough times, eventually you will get it right. A few of us have been in IT for long enough to relate to this personally. Bring on the renaissance, we're ready!

Delight, Satisfy, or Frustrate?

2005-10-24T18:40:00.000Z

My recent posts have discussed the relationships among user expectations, site responsiveness, and user satisfaction. To sum up the conclusions of all the research I have seen (and done) into Web download times, Web users’ tolerance for delays depends on several factors, including their expectations, site feedback, the complexity of a task, its importance, and the relevance (utility) of the information being provided by the site. But their perception of a site’s quality and credibility diminishes as its download times increase.

Today I am going to shift my attention to performance management. For an eCommerce company embarking on a program of Service Level Management (SLM), what are the implications of all this research? How should you set response time objectives? Here are my conclusions:

Treat Miller’s thresholds as a design framework to be kept in mind (more on this below).
Forget about Bickford's 8-second rule! Web norms have progressed since 1997.
What really matters are the service levels your customers expect. You can delight them, satisfy them, or frustrate them by delivering levels of responsiveness that they perceive to be fast, reasonable, or slow respectively.

Writing about the role of evolution in site design, Jared Spool observed:

"For most types of sites, there are existing models already out there. Someone who wants to produce a new site with pharmaceutical information, for example, can find dozens of existing pharmaceutical sites. A small investment in studying how users interact with existing sites can reveal a lot about what works for your users on their tasks. You could easily develop an understanding of the best practices and, from that, produce your own guidelines".

The same applies to your users' perceptions of performance, and how those should determine the objectives you set for your site. Because those perceptions will have been set by other Web sites (including your competition), the performance of comparable online services delivered by other sites in your market segment should be your starting point. You can never go wrong by measuring your competitors and striving to match them.

For example, Keynote's Public Services division publishes weekly indexes of Web site performance for several industry verticals. Many companies in these markets monitor these indexes as an important benchmark of the competitive climate, and of their own site’s quality.

Some leading companies have taken this approach several steps further, setting up comprehensive measurement, tracking, and reporting programs for their most important Web initiatives, and letting their measurements of the competition drive their own service level objectives. One company even recalibrated the annual performance objectives and bonuses of IT and development staff, based on their ability to match competitors’ performance. And it works! The company improved from last to first place in its industry by following this SLM strategy consistently over a 3-year period.

So if matching the competition is everything, how are Miller’s thresholds relevant? I see them as absolute standards within which you will frame your relative (competitive) objectives. Although you should aim to compete, you can relax a bit more if you are already close to meeting Miller's guidelines. For example, if your competitors can find widgets in 4 seconds, your site's 8-seconds response is too slow. But if they take one second and your site takes two, you are doing OK -- focus your tuning efforts on another online application that needs help. For additional perspective on this subject, I recommend reading this 2002 exchange in Business Communications Review between Peter Sevcik and Peter Christy.

A related aspect is the nature of the service your site is providing, in response to the user's input. Miller proposed that ideally all computer responses should occur within two seconds. But in practice, it is reasonable to absolve some complex interactions from this 2 second target, if you are sure that the customer would not expect the response in 2 seconds. Credit card validation is a good example. If in doubt, be guided by what other sites can achieve for a comparable function. The subject of Web "norms" for various types of interaction is one that I plan to revisit here. It is already being discussed in the context of the Apdex approach to setting objectives and reporting response times.

To sum up, performance matters! It affects your bottom line, if you are doing business online. And to win at eCommerce, you cannot be content just to match your competition at every turn. Look at the situation from their point of view. Their customers’ expectations will also be based on the prevailing Web climate. So if your site is faster, the competition can no longer delight their users with their responsiveness – the best they can do is satisfy them. This gives your site an edge in the usability stakes, which a competitor must now try to compensate for by doing better in some other aspect of their site. If they don't, and remain noticeably slower for an extended period, eventually some of their customers will become frustrated, and switch to your service.

So having a faster site gives you a competitive advantage. This is where the money you have invested in your SLM program really pays off. I've been working in this area recently, and in future posts I will be writing more about the value of SLM, and how to calculate return on investment (ROI).

WYSIWYG, or No Site is an Island

2005-10-23T21:30:00.000Z

When writing about Human-Computer Interaction (HCI), I discussed Robert B. Miller's classic research into computer responsiveness and its relevance today to questions of Web site design and site usability. For one response to Miller's findings (by providing more percent-done indicators and busy cursors) see Jakob Nielsen's Web site and his book, Designing Web Usability.

But Miller's three thresholds are far from the whole story. Some digging on Google will reveal that in the last 10 years, scientists around the world have investigated the effects of page download time differences on Web users. Why did these researchers , who were usually aware of (and often referenced) the earlier results of Miller and Bickford, feel the need to conduct new studies focused on the Web? Was it because the Web was reaching a new population with no previous experience of mainframes or hypertext systems? I don’t think this is a sufficient explanation, because Miller’s results are still applicable. I believe it was because they understood that people’s ultimate experience of, and response to, any computer interaction depends most of all on their prior expectations.

In his Law of the Internet User Experience, Nielsen pointed out that Users spend most of their time on other sites. In July 2000 he concluded that this fact, together with others, would lead to the End of Web Design -- meaning that Web site designs would converge. Five years later, it is clear that his controversial conclusion has not really come to pass.

While some common idioms and design patterns have emerged, most would agree that site designs still vary widely. In fact, in September 2004 we find Nielsen himself comparing the Web to an anthill built by ants on LSD. Writing about The Need for Web Design Standards, he observes that while users expect 77% of the simpler Web design elements to behave in a certain way, ... confusion reigns for many higher-level design issues.

I believe a key reason for this state of affairs is the presence of what Jared Spool labels design evolution, as sites adapt to observed user behavior. Evolution promotes both diversity, as novel traits emerge to take advantage of environmental niches, and convergence, as less efficient species tend to become extinct. Jared's conclusions about design evolution touch on both aspects:

A small investment in studying how users interact with existing sites can reveal a lot about what works for your users on their tasks. You could easily develop an understanding of the 'best practices' and, from that, produce your own guidelines. [Convergence]

Because you will generate your guidelines by directly observing your users, these guidelines are far more likely to be of value than generalized guidelines produced from sites that have little or nothing to do with your work. Evolution has produced these sites and you can identify which have won the 'survival of the fittest' competition. [Diversity]

In nature, the attribute of speed evolves in only one direction, because faster individuals more often capure their prey and escape predators. On the Web, human nature, specifically our impatience, drives up individual site performance. One consequence of Jakob's Law that he did not write about is that users' expectations of site performance must inevitably converge, because they are set by their experience of other sites. As sites get faster, people expect faster sites, and are less tolerant of slower ones.

While Miller’s findings identified some important (and largely invariant) behavioral thresholds that apply to all human-computer interactions, an individual person's satisfaction with their interaction with a Web site will always be determined by more than those thresholds alone. The key additional ingredient is their prior experience of the Web environment as a whole, which sets their expectations and provides the context in which they can judge subsequent online interactions.

Nielsen's latest candidate for the scrap heap is the old acronym WYSIWIG. So maybe it's OK to recycle it with a new twist: As a Web user, what you suppose is what you got. Or, as John Donne might have adapted his metaphors for today's interconnected world:

No site is an island, entire of itself
Every site is a piece of the Web, a part of the environment

The Miller Response-Time Test

2005-10-22T23:20:00.000Z

In the field of Human-Computer Interaction, HCI for short, one crucial and much-studied aspect is the speed of the computer's response to various kinds of user inputs. Although a few of these studies do get quoted in discussions of Web site design and Web usability, most books and articles on these topics devote very little space to this aspect.

One of the best summaries appears in Andrew B. King's book Speed up Your Site, which opens with the simple observation that People hate to wait. His first chapter (Response Time: Eight Seconds, Plus or Minus Two) includes a brief history of HCI research into the influence of computer response time on user satisfaction (or frustration), and an overview of its relevance to Web site usability.

In the early days of the Web, Service Level Management meant little more than keeping your site up and running. Then came the 8-second rule of Web page download time, as people began focusing on the need to build and maintain a consistently responsive Web presence. This rule originated in research results presented by Peter Bickford in his paper Worth the Wait?, published by Netscape in 1997.

For a while, Bickford’s paper was widely quoted whenever Web site performance was discussed, and the 8-second statistic soon took on a life of its own as a universal rule of Web site design. Bickford’s actual results however were more variable that this simple “rule” might suggest. Only half the test subjects abandoned after 8.5 seconds, and site feedback (like animated cursors or progress bars) kept them around for much longer.

Interestingly, the 8-second rule was by no means the first widely accepted rule in the field of human computer interface design:

In the world of hypertext systems (which predate the Web and HTML by about 25 years), Akscyn’s Law was well established prior to the first Hypertext Conference in 1987. It states: “Hypertext systems should take about 1/4 second to move from one place to another. If the delay is longer, people may be distracted; if the delay is much longer, people will stop using the system”.
As long ago as 1968, when all computers were mainframes, Robert B. Miller’s classic paper on Response Time in Man-Computer Conversational Transactions described three threshold levels of human attention. A response time of one tenth of a second is viewed as instantaneous, a response within 1 second is fast enough for users to feel they are interacting freely with the information, and response times must stay below 10 seconds to keep the user’s attention focused on the dialog (note that this is similar to Bickford’s findings). Miller also concluded that a consistent 2 second response would be ideal.

Many researchers have investigated the subject of Web responsiveness since Bickford, but no new universal rules have emerged. This is no surprise, because Miller’s findings were a direct result of how people’s brains operate, and they applied to any human interactions with machines. Changing the machine from a mainframe terminal to a Web site has not changed people’s brains.

So how does the Web experience today measure up to Miller's norms? Jakob Nielsen has been writing about this aspect of Web usability for over 10 years, and although he is still pessimistic about Internet technology in general (see The Need for Speed) the very best and most popular sites are actually doing OK these days. The home page download times of the top 10 or so sites in The Keynote Business 40 index do achieve Miller’s 1-second threshold (for users with a high-speed connection). Many more pages achieve Miller’s 2-second guideline.

But it’s still safe to say that the vast majority does not yet pass the Miller Response-Time Test of computer usability. So if your site's pages are slower, then you still have some work to do before you can feel assured that their design is ideally suited for human-computer interaction. Dogs maybe, but not humans.

Performance Matters for VoIP

2005-10-21T18:55:00.000Z

In his earlier post on Web Usability: A Simple Framework, Chris Loosley considers what requirements web sites must fulfill to satisfy users. The simple framework he proposes breaks down the principles of Web Usability into four distinct areas: Availability, Responsiveness, Clarity, and Utility. To attract and retain users, Web sites must satisfy these four needs.

As the VoIP industry rapidly grows and expands, it too needs to address issues of usability in order to attract and retain new customers. Performance matters when it comes to VoIP usability, and to succeed in the marketplace a VoIP service offering must satisfy the same basic needs as a Web site:

Availability is an obvious requirement. The traditional telcos have set very high expectations for the market. Using a phone on the Public Switched Telephone Network (PSTN), users expect to be able to get dial tone and place a call 99.999% of the time.

Responsiveness is a critical performance factor in VoIP service satisfaction. Every VoIP call has a certain amount of delay, measured in milliseconds, for the audio to travel over the virtual circuit between phones. Excessive delay causes conversational disruption that can quickly lead to user dissatisfaction.

Clarity in phone calls is also an important performance factor. As Chris describes it, clarity describes a service that is "simple and natural to use." A "simple and natural to use" VoIP call is one in which the audio sounds clear and natural. Calls need to avoid audio defects that get in the way of understanding the speaker on the other end of the line.

Once the other three needs are met, the quality of Utility addresses whether or not the VoIP service actually delivers the value, features, and customer service that the customer looks for. Is international dialing easy to do? Does the voicemail system meet the customers' needs? Is the billing process accurate and easy to understand?

Customer satisfaction in VoIP systems involves the same Service Level Management requirements that web sites must meet. Of the four qualities described above, three -- Availability, Responsiveness, and Clarity -- fall under Service Level Management. This is different from Web Usability, where Clarity is a more qualitative aspect of the total customer experience. For VoIP, great strides have been made in converting the qualitative assessment of how clear a call sounds to the human ear into a quantitative measurement that can be automated and managed as a service level performance metric.

Although VoIP service performance is measured differently from Web site performance, providers of VoIP services have just as much at stake in managing their service level performance. VoIP service level measurement and management are rich topics that I plan to cover in future posts.

The Application Performance Index

2005-10-20T04:21:00.000Z

Yesterday I introduced the subject of Service Level Management, or SLM. To manage application service levels effectively, and satisfy your customers, you must monitor and report on availability and response times. So if you collect 10,000 measurements, what's the best way to report them?

Availability percentages are the easy part; they tell a story that everyone can understand. If 5 of your measurements failed, then your availability for that period was 99.95%. All you have to do is report the overall percentage for management tracking purposes, and perhaps summarize the causes of the 5 errors for technical staff to follow up and see if those kinds of errors can be reduced or eliminated in future.

Aggregate response time statistics are not nearly as self-explanatory. Assuming that you have set response objectives for that application, statistics like average response times (or even averages with standard deviations or confidence intervals, for the statistically minded) do do not really show how well you are meeting your goals and satisfying your customers. While technicians may have the time to discover important patterns from frequency distributions and scatter plots, managers need a quick way to understand the bottom line.

This is especially true of Internet measurements, whose distributions can be so skewed that their average does not represent the “middle” of the data. In practice, a few really slow measurements can push up the average, so that as many as 85% of all measurements may actually have been faster than the average. This presents a challenge, especially if you have set different response objectives for many pages of your Web applications, and those pages exhibit different response-time distributions. Reporting that "average response time was 3.7 seconds" is not very informative.

How then should you summarize and report page response times? Until recently, there was no accepted way to reduce response-time data to a common scale that would immediately show managers the level of success being achieved through their SLM efforts. Apdex, short for Application Performance Index, is a new open standard that seeks to address this problem. An alliance of companies whose business is measuring performance has defined the Apdex metric, a user satisfaction score that can be easily derived from any set of response time measurements, once a response time goal has been set.

The Apdex specification defines three zones of responsiveness: Satisfied, Tolerating, and Frustrated. The satisfaction threshold (T) is your response objective, and the frustration threshold (F) is always set to 4T. This simple rule is justified by the empirical findings of usability research, which will be a topic for a future post. The Apdex metric is computed by counting the satisfied samples plus half the tolerating samples (and none of the frustrated samples), and dividing by the total sample count.

The result is a number between 0 and 1, where 0 means no users were satisfied, and 1 means all users were satisfied. For example, if there are 100 samples with a target time of 3 seconds, where 60 are below 3 seconds, 30 are between 3 and 12 seconds, and the remaining 10 are above 12 seconds, the Apdex score is (60+30/2)/100, or 0.75. This result can be reported in one of two standard formats: 0.75[3.0], or 0.75 with a subscript of 3.0. The key point is that any display or report of an Apdex metric always includes the value of the target T that was used.

So if you achieve an Apdex score of 1.0 by setting yourself the easy target of 25 seconds, your reports must show 1.0[25]. But if you use more appropriate Apdex thresholds that truly reflect the level of service you want your customers to experience, then your Apdex score will tell you how successful you are in reaching your own goals.

I believe the Apdex approach is a really good idea, and I will be discussing it further in future posts.

Service Level Management (SLM)

2005-10-18T22:31:00.000Z

Building and maintaining first class e-business applications demands a systematic commitment to delivering levels of quality that can be measured and managed. Companies must address many inter-related issues, including:

What level of performance do our customers really expect?

How can we match, even stay ahead of, the competition?

How will we prepare for our next big sales event (or season)?

How will we measure site and application responsiveness?

How will we know when our customers experience a drop in service levels?

How do we diagnose and fix problems quickly?

How will we monitor, quantify, and report on our success?

The management and technical activities required to tackle these issues are collectively called Service Level Management, or SLM for short. To implement SLM successfully, many people with diverse skills and responsibilities must contribute, because SLM touches every aspect of the application lifecycle -- site design, database design, application programming and testing, systems management, and networking.

This is a broad topic, every aspect of which I will probably be writing about here at one time or another. Having worked for years in the world of software and application performance, I tend to use the terms Service Level Management and Performance Management interchangeably. But these days, people without that background are more likely to interpret the word "performance" as a reference to Business Performance, or BPM for short. So unless the meaning is clear from the context, I will try to remember to use the SLM terminology.

If you would like to read a book about SLM for the Web, I recommend Practical Service Level Management: Delivering High-Quality Web-Based Services by John McConnell and Eric Siegel. As a friend and former colleague of Eric's, I am a little biased. But I think the Amazon reader reviews speak for themselves.

Web Usability: A Simple Framework

2005-10-18T05:47:00.000Z

Web Usability is a big topic. I just Googled "Web Usability" and got 2,350,000 hits. "Web Usability Guidelines" is slightly more manageable, with only 598 hits. There is even a widely referenced research paper devoted to A Framework for Organizing Web Usability Guidelines.

[This reminds me of a saying I learned as a college student: "Those who can, do. Those who can't, teach. And those who can't teach, teach the teachers." Maybe if I scour Google for long enough, I'll find a framework for organizing Web usability frameworks. But I digress.]

I own several books on Web Usability, and I've looked at a far greater number of books and Web sites at one time or another. It's not hard to find long reading lists like this one by Paul D. Hibbitts. This appears to be a good list -- but who has time for that much reading? I will try to assemble a much shorter list of recommendations for a future post.

Despite (or maybe because of) the widespread attention to this topic, there are many ways to slice the usability pie, and each author seems to offer a different taxonomy. But all the best books do have one thing in common: their weight! No matter who you read, they give you a lot of guidelines to follow. In fact, far too many for my liking.

Being a simple-minded mathematician at heart, I am always looking for those few principles that will provide a sufficient foundation for most of what I need to remember. So, since I'm mostly interested in the role that site performance plays in determining usability, here is my own simple framework. I believe that to satisfy customers, a Web site must fulfill four distinct needs:

Availability: A site that’s unreachable, for any reason, is useless.

Responsiveness: Having reached the site, pages that download slowly are likely to drive customers to try an alternate site.

Clarity: If the site is sufficiently responsive to keep the customer's attention, other design qualities come into play. It must be simple and natural to use – easy to learn, predictable, and consistent.

Utility: Last comes utility -- does the site actually deliver the information or service the customer was looking for in the first place?

Until a customer has established a reason to stay on a site, I believe this framework presents the four essential qualities in order of their significance, and therefore applies to all first-time visitors. Familiarity with the site can alter a customer’s priorities, but only a prior knowledge of some unique utility not obtainable from other sites can overcome frustrating slowness or poorly designed navigation features.

For my purposes, this is enough detail. For the time being, at least, adopting this simple framework saves me from having to attend Jakob Nielsen's tutorial to learn Which of the more than 1,200 documented Web usability guidelines are most important? But I would hazard a guess that about 85% of those guidelines, and those proposed by other Usability experts, fall under my heading of Clarity.

On the other hand, I don't believe that issues of Clarity represent anything like 85% of the challenges to be overcome to make a site truly usable. Such an assumption would not give enough weight to the challenges presented by the other three areas. It would also downplay the dynamic nature of the Web experience and the interplay between these four factors as sites, and users' expectations of sites, evolve.

In particular, I believe that anyone responsible for setting site performance goals needs to understand how customers' expectations are likely to be modified by their experience of a site, and of the Web in general. And understanding what customers expect is the essential first step in providing a satisfying Web experience.

Eventually I plan to tackle this subject at the next level of detail. But first I may need to read a few more of the books on Paul Hibbitts' list!

When is Your Web Site Fast Enough?

2005-10-16T21:10:00.000Z

When your business is your Web site, and site responsiveness affects customer satisfaction, how much to budget for improving site performance becomes an important business decision.

Companies doing business on the Web must manage both site availability and site responsiveness. It's relatively easy to justify spending money to ensure site reliability; when customers can’t reach your site, you're losing business. And even if you can’t assign a precise dollar cost to every outage, at least everyone understands why the ultimate goal is 100% availability, and why a 10% downtime record is likely to be ten times more damaging to the business than 1%.

A cost/benefit analysis of responsiveness is a lot trickier; how much should you invest in speeding up your site? Faster may be better, but how fast do you really need your site to be? And even if you have set a target, what is the cost of missing it? These are the harder questions facing eCommerce executives today.

I wrote about this topic last week in eCommerce Times. The article provides a short overview of a very important subject, which I plan to write more about in future posts.

Why Performance Matters

2005-10-16T19:11:00.000Z

In trade magazines and newsletters, articles featuring the “Top Ten Rules” are a staple feature. So anyone with even a passing interest in Web site design has probably seen a few checklists devoted to site usability. In essence, these lists present ways to keep visitors on your site --rather than driving them away in frustration. These guidelines are important for every Web site, but become absolutely vital when doing business online. Over the past ten years I’ve read many such articles -- and I have been delighted to see that they almost always list the importance of site responsiveness.

Why is this so pleasing? Well, having devoted most of my career to software performance engineering, I have always regarded performance as a fundamental cornerstone of software quality. But in the past, such concerns were often viewed as an arcane technical backwater. So it is gratifying to see that performance is now widely acknowledged as vital to Web site usability and customer satisfaction.

Software performance is a very large subject; I know, because I once spent two years writing a book (High-Performance Client/Server) about it. The book mainly describes timeless principles of performance engineering and how to approach distributed computing with performance in mind, but (since I wrote it during 1996 and 1997) the examples are a bit dated now. Its focus really needs updating to address the Web environment, but I don't think I'm going to find the time to do it.

Publishing a blog seems to be a better plan. I aim to contribute an organizing framework and a regular supply of ideas. And I also hope to keep things interesting by attracting comments and contributions from others. As Web users, we all know that site performance does matter, so I will try to make this an interesting place to discuss Performance Matters.

2005-10-16T07:00:00.000Z

[Last updated on 8/27/2006]

Foundations
Probability: The Rain in Spain ...
Waterfall Methods: Past and Ever-Present
Software Engineering Matters

Performance as an Aspect of Usability
Why Performance Matters
Web Usability: A Simple Framework
The Dimensions of Usability
Performance Matters for VoIP

Web Usability Books
1. Don't Make Me Think, by Steve Krug
2. Designing Web Usability, by Jakob Nielsen
3. The Design of Sites, by Douglas Van Duyne et al
4. Usability For The Web, by Tom Brinck et al

SLM Overview
Service Level Management (SLM)
Keeping a Public (Site) Healthy
Performance: Deja Vu All Over Again
The Web Site Availability Model
The Web Site Response Time Model
The Value of Reference Models

Managing Rich Internet Applications
White Paper Series:
    1. Introduction to RIAs
    2. SLM Issues for RIAs
    3. RIA Technology
    4. The RIA Behavior Model
    5. Measuring RIA Responsiveness: Introduction
    6. Measuring RIA Responsiveness: Complications
    7. RIA Usability and the Site Development Process
    Update: RIA White Paper and Wikipedia

Alistair Croll on Ajax
Reporting Web Application Responsiveness

Web Performance Engineering: Building in Performance
1. Ten Steps to a Faster Web Site, by Alexander Kirk
2. Web Performance Tuning, by Patrick Killelea
3. Speed Up Your Site, by Andrew B. King
4. Deliver First Class Web Sites: 101 Essential Checklists, by Shirley Kaiser

Setting Performance Objectives
When is Your Web Site Fast Enough?
The Miller Response-Time Test
WYSIWYG, or No Site is an Island
Delight, Satisfy, or Frustrate?

Capacity Planning and Load Testing

Measuring Performance
Deep Thoughts on Management
Yahoo! on Web Page Performance

Detecting, Diagnosing, and Fixing Problems
Keeping it on the Road

Reporting on Performance
The ABC's of Measurement Data
The Application Performance Index

SLM Cost/Benefit Analysis
The Business Case for SLM
SLM: Learning from Dot-coms
Armstrong on IT-Business Alignment
Climbing The SLM Maturity Ladder

Performance in the News
Insights from Interop 2006
Are Online Retailers Ready for Business?

Notes

This index will be updated to include references to all significant posts on performance subjects. Those posts may in turn have links to further posts on topics within their subject areas.

Although there is no editorial calendar, subject areas without links do suggest topics we hope to cover in future posts.