The World of Pain that is HTML5 Video
January 17, 2012 in Tech
With the continuing momentum of HTML5, industry expectations of the online video space continue to grow. Unfortunately, there are many misconceptions about HTML5 video, and I find myself in the position of explaining to my clients (many already heavily invested in Flash-based video solutions) that current HTML5 capabilities are not on par with those of Flash Player 9, released over five years ago, much less the current Flash Player 11. And so, I’m doing more of the same exercises I did when native video playback was first introduced in Flash Player 6, in 2002: managing my clients’ expectations of what can be achieved with current technologies. More importantly, when my clients say they want to support HTML5 compliant video, the subtext is, “I want my video to work on iPhones and iPads.”
Given the current state of affairs, I wanted to assemble some notes on the state of HTML5 video, as a helpful introduction to those of you who need to start conceiving, planning and building solutions that necessarily involve aspects of HTML5 video. I’ve had to consider all of these factors and more as I’ve built my own video encoding and hosting service at videoRx.com over the last two years.
Introduction to the Problems (or Pain)
Many web developers already believe that HTML5’s <video> support can do away with plugin-supported video delivered by Adobe Flash Player or other runtimes. My three biggest concerns with using HTML5 <video> for “real world” video deployment revolve around:
- Current cross-browser implementations and adoption rates
- Cross-platform mobile optimizations
- Codec support
| Note: There are more than three reasons to proceed with caution using HTML5 video. For example, video playback capabilities such as true full screen support with graphic and text overlays that online audiences have come to expect from online experiences delivered with Flash-based video players aren’t available in customized HTML5 video players. I’ll compare Flash and HTML5 video capabilities in a future post. |
Cross-Browser Implementations
I’ll start with current desktop browser usage statistics. Microsoft Windows is still the primary desktop operating system in use today. Microsoft has decided to only release Internet Explorer 9 (IE9) and higher on Windows Vista, Windows 7, and future versions of the operating system, despite the fact that 46.52% of the desktop market is still using Windows XP, which can only run IE8 or lower. The HTML5 <video> tag is only supported in IE9 or higher. And by most counts, Windows XP has several years left as a viable operating system. As Table 1 illustrates, nearly 30% of desktop browser usage in North America from December 2011 to January 2012 could not interpret the HTML5 <video> tag. By default, then, if you want to reach a significant portion of desktop viewers with browser-based online content, you need to have a solution that more than likely will require the Flash Player. If you don’t believe me, just look at YouTube, which still uses a Flash-based player as the default video player wherever Flash is supported—even when an HTML5 <video> ready browser is detected.
| Note: Windows XP can run other non-Microsoft browsers such as Mozilla Firefox or Opera that are HTML5 <video> compliant. However, as I later discuss, these popular browser alternatives do not support H.264 video playback. |
| Browser | Market Share | <video> support | Codec support |
|---|---|---|---|
| Desktop | |||
| IE8 | 24.83% | – | – |
| Chrome 16 | 19.83% | ✓ | H.264*, WebM, Theora |
| IE9 | 15.75% | ✓ | H.264* |
| Safari 5.1 | 5.9% | ✓ | H.264* |
| IE7 | 4.16% | – | – |
| Firefox 3.6 | 2.95% | ✓ | Theora |
| IE6 | 0.89% | – | – |
| Mobile | |||
| Safari Mobile (iOS) | 45.89% | ✓ | H.264* |
| Android Browser | 36.88% | ✓ | H.264*, WebM** |
| Blackberry | 7.33% | – | – |
* Hardware acceleration available
** Not available in early versions of Android browser.
Mobile Optimization
While the desktop browser market may not be HTML5 video ready, I frequently hear that one segment of the market is HTML5-ready and has been for sometime now: the smartphone and tablet markets. By and large, it’s true—nearly every Android and iOS device supports HTML5 <video> playback as shown in Table 1. But, HTML5 <video> options here ironically are not good enough for the mobile platforms that so desperately need enhanced attributes. Mobile devices require special video care: specific video formats (H.264 Baseline for widest support), and video formats that are hardware accelerated to avoid unnecessary battery usage. Cross-browser HTML5 <video> support is limited to progressive downloads, which aren’t exactly mobile friendly. Yet most video delivered on the web (and to mobile!) still use progressive download over HTTP as the primary delivery method.
Apple is the only browser vendor to introduce a specification, called Apple HTTP Live Streaming (or HLS), that’s available in all iOS and now some Android devices. But, this specification is not adopted by other major vendors including Microsoft, Mozilla Firefox, and Opera. Google is the only other vendor to implement HLS with the native Android browser, universally in 3.0 versions and with limited support (varying by device manufacturer and carrier) in 2.3 and higher versions. Another streaming specification that’s been introduced is MPEG-Dash, which has many promising mobile and bandwidth friendly features but is not yet adopted by browser vendors. However, a streaming protocol is just one aspect of being mobile (and desktop) friendly. The more important aspect of an HTTP streaming protocol is adaptive streaming, which enables the browser to determine the appropriate video quality (or bitrate) to play. Adaptive streaming requires the content producer to create two or more versions of the video per device target, each version having a specific bitrate that will play well for a given Internet connection speed. Apple requires any iOS application accessing video over cellular networks to use HLS adaptive streaming for any video playback within the application that is over 10 minutes in duration or over 5 MB of data in a five minute period (read more).
Codec Support
Lastly, there’s the video (and audio) codec issues with current HTML5 browsers. IE9, Safari Mac 3.1 and nearly all of the native smartphone browsers support the industry standard AVC/H.264 video codec, as well as the AAC audio codec. However, Firefox 4+, Opera 10.6+, and Chrome 10.1+ support the WebM video codec, a newly open-sourced video codec, and the open source Vorbis audio codec. (Firefox 3.5 and Opera 10.5 browsers also support an older open source video codec, Theora, which is not optimized for high quality video at lower bitrates as modern video codecs are.)
If you want to adhere to current HTML5 video standard(s) and not rely on plug-in based playback, then, you’ll need multiple versions of a video, each encoded with the codecs necessary across all HTML5 browsers. You’ll also need multiples of those multiples, if you want optimal mobile playback with adaptive streaming. Developers focused on “standards only” would have you believe it’s best to encode your video in four or five different versions. This approach adds considerable time and expense to any video distribution workflow for a content producer that has more than just a couple of videos to manage on their site.
Standards versus Business Goals
In many respects, a strict standards approach for online video delivery is not financially responsible for business. With the current state of HTML5 video, there are too many codecs to support, and limited options for delivery protocols. WebM is far from an industry standard, and we’re years away from seeing WebM hardware acceleration across devices and desktop that we currently enjoy with AVC/H.264. While HTML5 is “video ready” for smartphones, WebM as a viable codec option is not. Apple iOS doesn’t support WebM, and for reasons similar to Apple’s resistance to the Flash runtime, I don’t see Apple adopting WebM anytime soon. Simply put, if company X is producing the hottest and best-selling devices to the same folks you want to see your video content, then company X’s implementation is going to dictate how you publish that content. Right now and for the foreseeable future, company X, of course, is Apple. Apple and Microsoft have both rallied behind H.264 as the preferred video codec for video playback, from device to desktop to TV.
If you view the current online video landscape from a business perspective, you likely want to achieve one or more of the following:
- Maximum viewership of your video. You want your video to play across mobile and desktop browsers with a consistent experience as much as possible.
- SEO (Search Engine Optimization) friendliness. You want your video content recognized by as many search engines as possible.
- Efficient encoding workflow. Video, unlike other HTML visual assets, requires substantially more time, effort, and storage. You want a manageable workflow that can reuse assets across delivery mechanisms.
- Wide range of delivery options, from live streaming to protected content. Business stakeholders will increasingly want to monetize their online video content, and HTML5 video doesn’t provide methodologies to limit access to the video source files. Apple HLS and MPEG Dash are the only options to stream and encrypt content, but only the former option is available now and only on iOS and limited Android devices.
The following factors should not, and likely will not, influence the business of online video:
- Standards compliance. In the best of all possible worlds, we’d see a simple <video> tag that works consistently with the same format and delivery options everywhere. We won’t have that option anytime soon. First and foremost, sites that have video content want their target audience(s) to be able to watch the video with as little fuss on both sides of the equation: reduced encoding and deployment effort for publishers, and reduced playback headaches for viewers.
- Open source reliance. While WebM and Vorbis are open source codecs, the industry standard for high quality video production is H.264. It’s consistently used for video from professional video capture to Blu-ray discs to online video. Just as the video disc market couldn’t support HD DVD and Blu-ray simultaneously (or Beta and VHS), there’s likely only going to be one winner in the online video codec battle. While H.264/AAC codecs can incur royalty charges for subscription and pay-per-view use cases, the vast majority of online content is under the royalty-free clause of MPEG-LA‘s licensing terms. And if you’re lucky enough to be one of the few content creators that successfully monetizes their video online, your royalty obligations for H.264 usage to MPEG-LA won’t prohibit you from being successful.
| Note: As a quick example, Louis CK’s online-only sale of his standup special for $5.00 would likely only incur a royalty of $0.02 per purchase. For 100,000 copies sold (or $500,000 in gross revenue), he’s looking at $10,000 in licensing fees for H.264 usage. Would he have been so successful with his venture had he only offered a license-free WebM version instead which had limited playback support? (For more information on licensing terms of H.264, read this PDF on the MPEG LA site.) |
A business goal-oriented approach deals with the problem of viewership and what’s widely available for your playback requirements. Facts support that Flash Player is very viable on desktop and on browsers that don’t play nice with the predominant (read: Apple) mandates of H.264.
H.264, HTML5, and Flash: A Solution to All Online Video Problems
So, here we are. We have HTML5 standards telling us to create three versions of our video (H.264, WebM and Theora) or more if you’re using adaptive streaming and/or H.264 profiles and bitrates specific to smartphone, tablet, and desktop deployment. We have a significant portion of browsers in use today that don’t implement the <video> tag at all. We have a perception that the Flash Player is on its way out, and that HTML5 can do most if not all of the things that Flash can do. The solution to the problem is relatively simple: base your solution around H.264 and forget about supporting WebM and Theora. If you consider yourself a professional web solutions expert and want to offer your clients (and, more importantly, your target audience) the best mobile and desktop experience, you’ll also encode and deploy for adaptive streaming on iOS, Android (where supported), and the Flash Player.
What an unfortunate mess of confounding options that we solution providers must present to decision makers. In my next post, I’ll outline the simplest and most effective solution to reduce the pain of distributing your video online.
Great article Rob! Don’t you feel there are encoding solutions on the market that simplify the transcoding of a single asset to multiple formats, bitrates or even the segmentation of assets for adaptive bitrate streaming? I’m not saying this removes all they hurdles for the tag, but may eliminate the transcoding and asset management side of things. You still have a need for multiple players if supporting Flash as well, and I agree you do support both, at least for now. Once again, great Post!
Thanks for reading the article, Tim! I’ll be talking more about your questions in the next article, but the short answer is yes–there are most definitely solutions that simplify adaptive streaming (I’ve built one!), but regardless of asset production simplification, the limited capabilities of non-H.264 and non-streaming HTML5 browsers are not worth supporting when there are still appropriate alternatives like Flash Player on desktop. Rally behind H.264 (and adaptive streaming when appropriate), and build the player set required for that deployment strategy. Don’t get into a trap of supporting progressive download “only” formats like WebM. When we see adaptive streaming hit WebM (and appropriate browser support for it), I would consider it a viable mobile streaming format and codec.
I’m sticking to animated gif videos. The way I do it I load the next on right before the previous one runs out of frames. It’s like changing the reels.
Nice post. I’ve pretty much settled on sticking everything on Youtube and letting them sort out the pain for the moment
Adam, thanks for the comment. I’ll be talking about how YouTube fits into an online video marketing strategy in the next post. Stay tuned!
[…] are especially pleased to launch with posts from Robert Reinhardt and Joseph Labrecque, as well as from our Senior Software Architect, Omar Gonzalez, and myself. You […]
Interesting! I recently did a project for a client that wanted html5 video with flash fallback. I went through all fases you explain in your post
I went for mp4 and webm formats. So that covered everything they needed.
I checked if formats are playable and if not go to the flashplayer.
30 video’s in 2 formats was enough for me and the server space
Good post!
Thanks for the comment, Karel. Did you do two versions of H.264 (Baseline and Main) for smartphone and tablet/desktop playback? If just one, which profile did you use?
I think you’re making a mountain out of a mole hill. Breathe, calm down…
Everything is H.264 or becoming H.264 compliant. Anyone who looks at your own table above sees H.264 is clearly winning. The only codec issues really come when you actually care about a browser that has <5% market share. Through normal Windows updates, H.264 is in IE8.
It's the whole "Blue Ray" vs "H "-something see I can't even remember.
I understand your point, just like all the flash developers out there. It's painful to have wasted so much of your career studying and learning a misfit technology that slowly but surely can no longer be used and no longer get paid for. I can probably find someone on craig's list that will give you a hug. 😉
Hi Rob:
I appreciate you taking the time to comment, but I’m sorry you felt the need to respond to Robert’s post with such a condescending tone.
If you are comfortable with all the variables and moving parts of HTML5 video, that’s great. That’s excellent. Give yourself a well deserved pat on the back. Heck, take the rest of the day off.
But, you should then recognize that you are in the very, very tiny minority. There is a tremendous amount of confusion in the space right now (and yes, a lot of that is in the Flash community, because those are the people who have been working fluently with video inside of browser experiences for major firms and brands for over 10 years). And, because of the confusion, firms are spending a ton of money on solutions they don’t understand, to achieve goals that may not actually be possible with current technologies. That’s the real world today.
And Robert very generously donated his time and energy to writing a very informational post to helping the community sort this out.
The number of views and tweets on this post speak to the value the community sees in this information, regardless of how much you’d care to dismiss the message, especially given the flimsy, immature and ad-hominem basis on which you’d attempt to do so.
Rob, I don’t see my points as making a mountain out of a molehill. It’s the rest of the web community that seems to embrace whatever information they’re given by the HTML5 community. The appropriate analogy is that many people think they see a small bump coming out of the ground and presume it is indeed a simple molehill, but when they’re finally up close (i.e. knee deep in the muck of a real-world project with video), they were actually seeing a mountain and underestimated costs for deployment. I see a lot of non-optimized video experiences on the web popping up, especially on mobile–largely due to the fact that video isn’t encoded or deployed properly. Yes, I hope you’re right that “everything is H.264” but I think the folks at Google, Mozilla, or Opera won’t agree with you. And, even in a blissful world of H.254 compliancy, the vast majority of H.264 content that should be adaptively streamed is delivered as an HTTP progressive download. (Note that there is content that doesn’t necessarily require adaptive streaming, but nearly all online video content benefits from adaptive streaming.)
Also, I don’t see my career in any way wasted on past learning. Flash technology has enabled me to work on some of the most exciting projects I could hope to be a part of: an Emmy winning full episode streaming player for ABC.com, applications that professional athletes use to evaluate their visual prowess, live streaming platforms that let surgeons share their expertise with other surgeons around the world, just to name a few. I started my career in the film industry (see http://www.imdb.com/name/nm1217725/), and Flash played very much into the video encoding and deployment expertise I’ve built since the late 90’s where producers created multiple videos (Real Player, Windows Media, and QuickTime versions). Who liked that setup? The only difference current HTML5 standards present today is that the user might be shielded from making decisions about which format to view, but the onus is still on the content producer to navigate a mess of implementations. (Brightcove just released a great white paper also talking about the inconsistencies of HTML5: http://go.brightcove.com/forms/en-html5-video-performance?c=70130000000jjee&p=70130000000jjee) Flash provided a number of ways for content producers to easily get their video online with a consistent experience.
“Everything is H.264 or becoming H.264 compliant.” http://hsivonen.iki.fi/webm-share/ would suggest otherwise.
Heck, I’m still waiting for Flash’s video capabilities to catch up to Director….
Thanks for the comment, Darrel! How’s the weather down in Portland today? I know Director will (likely) have the upper hand with low-level manipulation of video, and I know you get called in for crazy customized uses. What capabilities do you miss in Flash video that you use in Director?
[…] http://transitioning.to/2012/01/the-world-of-pain-that-is-html5-video/ […]
Thank you for the well thoughtout comments. It was very helpful and informative.
Great article Robert.
Your analysis is spot on.
Throw Premium Video and mandatory DRM support into the mix here and the outlook gets even worse!
Robert- Thank you for this excellent post and the research that went into it. I recently attempted to update my online video portfolio with the misconception (as you discussed in your article) that HTML5 video would solve all my browser compatibility problems. It really just revealed more challenges that I would have to overcome with encoding video using specific codecs for specific uses, and then properly coding multiple files into my website. This proved to be tricky and not foolproof, considering that it’s unrealistic to check playback for browser compatibility when you don’t have access to every OS, browser, Android phone, or iPad at your disposal for testing.
The breakdown you gave basically summed up my own weekend of a “world of pain,” where I came to the conclusion that it’s okay to stick with H.264, tinker with HTML5 video, and incorporate Flash. I still have to update my website, but I probably won’t be relying on HTML5 in the way I originally thought I would.
If I were (and I am) interested in making money for me (and even value for my clients)… then it is best for me to advise “leave flash now… go away, you losers–learn HTML5 dummies”. That way I’ll take on all the sloppy seconds or leftover crap that, if maintained, can provide lots more value for clients–and money for me. Yes, leave Flash. Go. Let me do all that’s left over. Thanks
let me add, I’m also available for HTML5, JavaScript, CSS work… in fact I suggest clients purchase the following:
–entire product built in Flash as a fallback on older browsers
–build the whole thing again for modern browsers in JS, recycling as you can and sharing data sources etc., but basically build it again.
Alternatively, some clients choose to attempt to serve every browser ever created–up to a point and then they write off some %. It’s just way less work to do it in Flash and reach nearly ALL of those stragglers (older browsers). Plus, building it out in Flash serves as a quick way to get sign off.
[…] with Flash as a web delivery medium for the past decade (see Robert Reinhardt’s earlier post on The World of Pain that is HTML5 Video). Browser Support for HTML5 Video Tag […]
Great article, congratulations!
There are two glaring problems here:
1) Firefox 4 is not listed in your market share table. It has more market share than half the browsers in that table. Why isn’t it listed?
2) Google announced that Chrome is planning to drop H.264 support. That means you need a really big caveat on Chrome’s H.264 support.
Additionally, Chrome doesn’t have full hardware H.264 decoding support. It can accelerate a few parts (scaling and YUV conversion). So can Firefox 4.
Thanks for the comment, Robert. RE: #1, you can click the links (“Desktop” and “Mobile” are linked to the data sources I used). I wouldn’t deny that some statistics report Firefox 4 having more market share, but I only reported North American browser market share, as most of my clients and their customers are North America-based. Even if it did have more market share, my argument about Flash covering that market share would still apply. RE: #2, it’s been over a year since Google’s announcement, and they’ve made no changes to date in the versions that have been released since then. But even if they did drop it, again, fallback to Flash. I think it’s critical to pick one codec and create your solution around that codec. Apple won’t be implementing WebM anytime soon—who’s more important, Firefox or Apple iOS users? Most of my clients would pick the latter.
Finally had the chance to have a good read or twice
of your post, Robert. I was experimenting the other day with HTML5 video and indeed felt the pain you’re describing here. And then my video was pretty darn simple… just a fullscreen moving background blur-effect with a little help of jQuery, so I can imagine what it must feels like if you need to make a full video experience for a client on an iDevice.
The time is not there yet. There’re still way too many bumps in the HTML5 video road. I think instead of saying we’re going to do a HTML5 video with a Flash fallback solution, we indeed better can say we’re going to make a Flash video with a HTML5 video fallforward solution.
Great article and thanks for sharing all the research you put into this.
[…] streaming which Flash owns for desktop and HLS owns on iOS. More info about the WTF here and the OMG […]
[…] var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(po, s); })();In my last article, I discussed the confusing landscape of options in online video delivery. Now, I want to outline an […]
Great post. Thank you for taking the time to write the post.
Robert, many thanks for this thorough post! It confirms some of the issues I have discovered while making the transition from Flash to the world of HTML5. I have been building a html5 video gallery for my company, and running into cross-browser compatibility issues. Specifically, I have found that apple devices are finicky about dynamically loading a new video into an existing element (on user selection). Have you also run into this issue and do you have a good solution?
It seems as though you may have abandoned Portland for Bainbridge? I will miss your contributions at local Adobe meet-ups. Thanks again for taking the time to share your findings! It’s very helpful to have someone with so much video expertise weighing in on the subject! Best of luck in your new hood!
Rebecca, thanks for the comments. RE: loading into an existing <video> element, I do recall that when I was building the first version of my videoRx JavaScript library that iDevices (at least the current iOS system at the time, two years ago) was having difficulty change the src attribute of an existing <video> tag. Initially, I wrote my JS lib to have a more SEO-friendly <div><video> hierarchy:
<div id=”videoContainer”>
<video id=”innerVideo” src=”etc..”/>
</div>
and then targeting the videoContainer div with swfobject for replacement with a Flash video player SWF (if Flash was supported), or swapping out a dynamic URL to an HLS adaptive stream for iOS in the innerVideo element. If I recall correctly, the innerVideo element would dynamically change on iOS consistently, so I just removed the <video> element from the static HTML and dynamically created it if Flash was not supported on the browser. I’m not sure if that’s helpful and/or confirming your own finding, but that’s what I remember.
RE: Bainbridge Island, well, I didn’t desert Portland willy nilly. My wife started an MFA program at Seattle University this past fall, and we moved a bit farther north to reduce commute time. We’re currently renting our house in NE Portland off Alberta Street, so who knows! We may be back in the future.
Thanks for the suggestions Robert. I’ll try ditching the static video element and see if using dynamic generation only helps.
Good luck to your wife on her MFA program! Pretty cool that you have easy access to SAM and the Frye Museum.
Is it possible to use the streaming video from HTML5 with multiple switches… choosing which stream to display based on user input?
I understand the codec issue, but if the codec is supported, and HTML5 is supported in the browser… whats the hang-up?
Also, the issue with XP… wow, is HTML5 really so crippled that it will not work on an XP machine? I find that hard to believe… but this is a new subject to me.
The HTML5 is still under development as of now.
As we all know that the HTML5 video capabilities is not as powerful as Flash.
It is a big step forward though, as in the future we can play videos directly on browsers without any plugin, such as the Adobe Flash Player.
Hi Robert,
Thanks for this interesting read!
I understand you prefer an industry standard codec (h.264), over a codec that is still in development (WebM). But aren’t you afraid of the power you would be giving to one company if the entire world kneels it’s patented codec?
Wouldn’t you instead prefer an industry standard, well functioning AND open source codec?
Thanks again!
Edo
Edo, thank you for taking the time to comment. From a technical perspective, VP8, as a video codec, is pretty awesome. It’s more a question of maximizing your audience with video assets that are the most compatible with browsers and plug-ins available. My clients need solutions that work today, not a year (or more) into the future. Plus, we can’t even stream WebM to browsers yet (MPEG-DASH). If you’re not maintaining a growing collection of video content, WebM can have a place in deployment workflows today—marketing videos, non-monetized content, and so on. H.264 though is a much more compatible video format for virtually all online distributions, from in-browser to offline/download.
[…] While HTML5 video works on all platforms, it works differently. Different browsers support different video codecs and differing implementations of the HTML5 video specification. Certain features that have been simple in Flash, such as adaptive streaming, are inaccessible in most HTML5 environments. Further exacerbating these pressures, browsers and device operating systems are updated on a nearly continuous basis, meaning that one must always remain current with a complex array of platform-specific performance capabilities. A consultant whom we bring on for many of our video projects, Robert Reinhardt of VideoRx.com, went into some detail on this point in a post earlier this year, provocatively entitled The World of Pain that is HTML5 Video. […]