The Era of the “Adaptable GUI” in Web Apps

September 11, 2012 in Business, Tech

Executing complex projects for the web has become more complicated and expensive a process in 2012, than it was two or three years ago. There are many drivers for this increase in cost — some of which have already been discussed at length on this site. One such driver of increased costs is the amount of thought, creativity and engineering that must go into the process of creating what we have come to call (for lack of a better term) adaptable GUIs or interfaces for complex browser apps.


Adapting to the 2012 Web

This process of adapting interfaces is not the same as porting experiences to new platforms. So-called platform extensions involve porting existing experiences to new platforms — the process of, for example, converting a website into an app for the iPhone or Nintendo Wii.

Netflix on the Nintendo Wii

No one expects Netflix on the Wii to present the same experience as Netflix in a desktop browser. This is a platform extension, rather than an adaptable GUI. Users do, however, expect Netflix to offer the same experience in IE and Chrome.

When executing a platform extension, one generally builds a new piece of software from scratch, designed to exploit unique characteristics of the target platform. Netflix on the Nintendo Wii is, for example, a platform extension of Netflix. No one expects Netflix on the Wii to present the same experience as Netflix in a desktop browser. People use desktop browsers and their Nintendos differently, in different contexts, with different peripherals. Users expect the experiences offered by platform extensions to be different — if they weren’t, they’d be poorly designed.

While platform extensions render experiences differently on new platforms, adaptable GUIs render experiences as identically as possible, within different configurations of the same environment — in this case, the browser. The purpose of an adaptable GUI is to present as consistent an experience and feature-set as possible, despite challenges in implementation and support of HTML functionality.

Ensuring cross-browser compatibility of one’s work has always been an issue in web development.

Ensuring cross-browser compatibility of one’s work has always been an issue in web development. Today, the process is something much more complex and intricate.

Of course, ensuring cross-browser compatibility of one’s work has always been an issue in web development. But today, in 2012, as more and more functionality is rammed into the browser, these issues have become significantly more burdensome to production efforts. Video-based projects, in particular, are ones in which the challenges of adaptable GUIs are readily apparent.


A 2012 Video Player

Let’s take the example of a basic video player intended to run inside of desktop and mobile browsers. This simple player will have only three features:

  1. adaptive streaming video delivery;
  2. the ability to pause and resume the video;
  3. and full-screen functionality.

While today YouTube offers multiple types of video players, including HTML5, YouTube launched, and ran for years, offering only Flash video.

Back in 2008, this would have been very simple. There was really only one technology that people used to display video inside a browser — Adobe Flash. HTML did not support video playback, almost all viewers had Flash installed, and it was a quick download for those who didn’t (much smaller than any other video playback plug-in). And, though Flash video didn’t really work on mobile, mobile was not a significant chunk of anyone’s market.

In 2008, building a Flash video player with the above three features could literally take as little as an hour, and possibly up to a few days — and (depending on your user base) more than 9 out of 10 visitors could immediately watch, without downloading any new software.

Fast forward back to today, and building a ‘Flash video player’ is no longer an option. Right off the bat, you will necessarily eliminate all iOS users, and an increasing number of Android viewers. Building video players in Flash is necessarily going to omit a large, and growing, segment of your potential viewers.

Thus you will want to utilize HTML5 for in-browser video delivery.


Catering to a Fragmented Market

Fantastic! We have an answer. Flash may not work for us anymore, but we have HTML5 video. But this satisfaction is short-lived, as we come to realize that HTML5 video is not going to solve all of our problems.

HTML5 Video Support

According to Long Tail Video, 74% of the market can play HTML5 video (up from 66% in the last quarter, and 50% from a year ago).

While relying on HTML5 for video delivery may work for 100% of your mobile viewers, many desktop viewers will be left behind. According to Long Tail Video’s continually-updated census, 74% of the market can play HTML5 video (up from 66% in the last quarter, and 50% from a year ago; although, the conflation of mobile and desktop in those statistics obscures how many desktop viewers still can not view HTML5 video).

Thus, somewhere in the neighborhood of one-quarter of your viewers can not watch HTML5 video. That’s an awful lot — nay, an unacceptable number — of viewers to leave behind. So we see that HTML5 alone can not meet our needs. What can you use to fill in those gaps?

Well, there’s really only one option… Adobe Flash. Using HTML5 and Adobe Flash together, to deliver video inside of a browser, ensures that almost 100% of your viewers will have immediate access to your video content inside a browser, on desktops, smart phones and tablets.

So it now seems that we have a clean answer. We’ll just use both HTML and Flash video, to ensure that our content is accessible to our viewers.

Unfortunately, it turns out that integrating multiple technologies to deliver a seamless experience across various browsers is a bit more complicated than it may appear on first glance. Indeed, there are a series of challenging trade-offs that must be addressed, when specifying even a simple video player that exploits both HTML and Flash for video delivery.


Feature Implementation

Central to the challenges of building successful adaptable GUIs is the fact that HTML5 and Adobe Flash do not support all of the same features — this is particularly the case with respect to video playback. Heck, not even all the browsers that support HTML5 video, support all the same features. Thus, there are certain features that will only be available in certain environments.

HTML5 Streaming Support

Today, only iOS reliably supports adaptive streaming in HTML.

For example, adaptive streaming (streaming video at a quality level based on the viewer’s connection speed, detected on an on-going basis throughout video playback) has been commonplace in Flash video players for years; today, however, only iOS reliably supports adaptive streaming in HTML. It is not possible to deliver adaptive streaming HTML video in, say, Internet Explorer, Chrome or Firefox.

Another such feature is full-screen video playback, which Flash has supported for several years. Yes, full-screen mode is part of the HTML5 specification, but (again, per the Long Tail Video census) approximately half of your viewers will use a browser that does not support full-screen.

Thus, of the three features I’ve noted as requirements of our video player (listed above), two — full-screen and adaptive streaming — are not cleanly implemented across the multiple technologies we are planning to use. The inconsistent availability of these features creates the potential for a complex or confusing user experience.


Feature Consistency and Availability

Given that some of our features are impossible in some of the environments in which viewers access our player, we essentially have three options:

  1. Remove the Inconsistently-Available Features
    The simple answer, from an engineering perspective, is to enable only those features supported by the current technology. So, for example, if someone is viewing the HTML version of our video player, in a browser that does not support full-screen mode, we simply do not render a full-screen button. In other words, we totally remove the inaccessible features from the GUI.
    While this is a simple answer, it is also quite a poor one from the viewer’s perspective. The same viewer may access our player from IE 9 at work, and Chrome 22 at home. This viewer would likely be quite surprised — and frustrated and confused — to see features added to and removed from the interface at different times of the day. After all, we’re not talking about two totally different devices; these are just two different desktop web browsers, and users have become accustomed to having consistent experiences offered in different desktop browsers.
  2. Message Unavailability
    Another alternative is to message the lack of availability of certain features. For example, for those users viewing our player in a browser that supports HTML-video, but not HTML full-screen, we could simply render the full-screen button in a disabled state (e.g., grayed out).
    If the viewer then attempts to interact with the button (say, by mousing over it) we could display a message explaining that the user is viewing the player in an old browser, and if they wish to use full-screen mode, they should upgrade to a newer technology.
    (It is important to remember that, while we can know what browser the viewer is currently using, we can not know what other browsers the user may already have installed. In these instances when features are not supported in the viewer’s current environment, we are also largely unaware if the viewer may have the required browser already installed on their system. For this reason, our messaging can not say definitively whether the viewer must download new software, or switch to another browser already installed on their computer.)
    This option works, resolving any confusion that would have emerged from simply ripping the full-screen button out of the GUI. However, forcing viewers to think about browser technology (and implying that they must download new technology to fully enjoy the experience) has the potential to intimidate certain viewers.
  3. Auto-Swap Tech
    The ideal solution is to have the interface respond, automatically and invisibly switching technologies to enable to requested feature. So, let’s once again return to our example of the user viewing our HTML video in a browser without full-screen support. Instead of graying-out the full-screen button, we could enable that button. If the viewer clicks the full-screen button to enter full-screen mode, we could automatically swap from HTML video playback to Flash video playback, and send the player into full-screen mode.
    While this may be the ideal option, it is also significantly limited. Recall, we can not know what other browsers the viewer may already have installed. Thus, in our example, we can auto-swap between Flash and HTML for certain features, but we can not switch between different browsers that may already be installed on the viewer’s computer. So, this solution will only work for some, but not all, of our features.


As this last point indicates, these options must be evaluated on a feature-by-feature basis. The solution you devise for full-screen mode, may not be the same one that you wish to pursue for support of adaptive streaming. Each feature that is supported inconsistently across your target environments requires individual consideration for the proper cross-environment treatment.


The Dividing Line

Using both HTML5 and Flash

Our player must utilize both Flash and HTML for video playback. So, the rest of the GUI can be either Flash or HTML or both — it’s up to us.

Our player must utilize both Flash and HTML for video playback. But that does not speak to the rest of the interface. Of course, a Flash GUI can control Flash video playback, and an HTML GUI can control HTML video playback, but the opposite is also true: a Flash GUI can control HTML video playback, and HTML can control Flash video playback. So, the rest of the GUI can be either Flash or HTML or both — it’s up to us.

Of course, we could just rebuild the entire player in both Flash and HTML — so that we are, in effect, creating two, separate video players. But that’s a lot of extra engineering effort, both now (as we build this first version of our player) and in the future (as we maintain our code, and add additional features). This is feasible, but comes at a significant cost (basically, we’d have to build everything twice) and will be seen as an undesirable option in many cases.

Well, if we don’t want to build our player twice, then we could build the GUI in either Flash or HTML. But, as we’ve already discussed, building the GUI just in Flash prevents our player from working on mobile, or on desktops without Flash player installed. So, it seems that the smart option (in this generic example) is to build the GUI in HTML, and have that GUI control video playback in both HTML and Flash.

At least we have a relatively clean resolution here. Except for one major issue: full-screen. Returning back to the earlier issue of full-screen behavior, let’s say we want to support Flash for full-screen video playback in older browsers (that do not support HTML full-screen). Building the GUI in HTML, but relying on Flash for full-screen mode, creates two annoyances:

      1. We can not display our HTML GUI in full-screen mode — our player can only display Flash content while in Flash’s full-screen mode. So, if we build our GUI in HTML, and wish to utilize Flash’s full-screen mode, we will have to rebuild our GUI in Flash, as well. (Meaning, we will have to build our GUI in both Flash and HTML, after all.)
      2. For security reasons, Flash full-screen mode can only be triggered by a mouse or keyboard click inside of the Flash movie. This means that, even with an all-HTML GUI, if we use Flash to deliver our video, then we must ensure that the full-screen button (part of the HTML GUI, recall) is actually a Flash element. In other words, even once we’ve resolved on an HTML GUI, at least one element of that GUI (the full-screen button) must be Flash.

Alternatively, if we do not support Flash’s full-screen mode, then we will only offer full-screen functionality to approximately half of our viewers.

This is fun, yeah?


Tech Defaults and Upgrade Paths

Ok, we’ve answered all of these internal questions so our awesome new video player is designed and ready, utilizing both Flash and HTML technology to deliver a fully-functional experience to all of our viewers.

Now, a viewer loads the player. This viewer has the ability to view both the HTML and Flash versions of our player. Which version should the viewer see?

Confusing Flow

A viewer with the ability to view both HTML and Flash loads the player. Which version should the viewer see?

Perhaps we wish to default viewers to HTML for video playback. Fair enough, but then we must be ready to sacrifice adaptive streaming in the majority of our player instances (as only Safari supports HTML adaptive streaming). Perhaps we want to default our desktop viewers to the Flash version, while sending mobile viewers to the HTML version. Perhaps we only want our desktop viewers to see the HTML version if they are in a browser that supports HTML full-screen mode.

As you can see, the question of default technology selection is one that is complicated by many of the considerations discussed above. With these questions, there are no right or wrong answers. You just have to decide on the defaults, and then realize the implications in user experience that will result. As different technologies offer different features and functionality, choosing which viewers encounter which technologies is a core determinant of the resulting user experience.

Related to this is the question of upgrade paths. Let’s say another viewer attempts to load our player, but this viewer is in IE8 (which does not support HTML video) and Flash Player 7. So, this viewer does not have any technology installed to view our video player. What do we tell this viewer? Should he upgrade his browser? Or his Flash Player? And is this influenced by any constraints of our viewers (e.g., if we cater to enterprise users with limited ability to install new technology on their computers)?


Technology Transparency

We have already discussed ways in which we might wish to automatically switch the viewer from Flash to HTML, or vice-versa, in order to provide seamless feature support in multiple environments.

Do we want to offer our viewers the ability to manually switch player technology? E.g., do we want to allow a user viewing HTML video, to intentionally opt to view the Flash video instead?

Transparent Technology

How transparent do we want to make our rather complex technology of the adaptable GUI?

On the one hand, we’ve gone to all this trouble to build our player in multiple technologies; why not make these different options accessible to our viewers?

On the other hand, the entire point of this endeavor is to provide invisible support for a consistent cross-browser experience; actually allowing users to switch between technologies detracts from the simplicity and transparency that is our intent.

I do not believe there is an obvious answer here — it depends on your priorities. If you wish to opt for simplicity in the user experience, then I believe you should not provide viewers the explicit ability to switch back and forth, from Flash to HTML for video playback. If, on the other hand, you wish to provide more flexibility to your viewers (and, your viewers are not the type who are intimidated by terms like ‘HTML’ and ‘Flash’), then perhaps it makes sense to provide this option to your viewers.


A Continuing, But Solvable Mess

Rodin's The Thinker

Production talent in 2012 is encountering a new set of issues and challenges: creating consistent and accessible experiences, across multiple browsers and devices, through the integration of multiple technologies in a single codebase.

Production talent in 2012 is encountering a new set of issues and challenges: creating consistent and accessible experiences, across multiple browsers and devices, through the integration of multiple technologies in a single codebase. These are not simple challenges. Feature support, functionality, user experience, business goals and engineering considerations must all be considered together, as one approaches a series of complicated trade-offs. This is all leading to the need to consider the design and production of software GUIs for web apps differently today, than before.

The notion of adaptable GUIs — having applications adapt to their environments, at such a detailed and subtle level as discussed in this post — is essentially brand new. Before the current era of fragmentation and mobile devices, Adobe Flash provided a consistent platform across browsers — none of the questions discussed in this post were relevant. Prior to the era of Flash, users just did not expect this type of cross-platform consistency (and it was essentially unattainable for all but the most vanilla of experiences).

True consistency in functionality, behavior and experience remains the goal (as it has been for years), but for complex in-browser experiences, it is a goal that is becoming increasingly unattainable. The number and variety of devices continues to expand, and the resulting mix of technologies is even more complex today than six months ago, much less three years ago. Instead of building a truly consistent cross-browser experience, we approximate it as best we can, through a series of trade-offs, custom tailored for our audience, balancing their preferences and usage patterns, along with our business goals.

Today, these issues are readily found in the HTML/Flash divide (particularly in the realm of video). How long this era of adaptable GUIs will persist is unclear to me. It is possible that it is a by-product of the results of the Flash/HTML war of the past few years, and will fade away as those battles finally conclude.

I suspect, however, that the market fragmentation we are seeing (that is the root cause of these new complexities and costs) will continue to expand, and with it, so will the need for consideration of adaptable GUIs, as we designers and developers seek to make invisible the browser-and-technology-induced discrepancies that would otherwise impede the user’s experience.