HTML5 in the browser: Canvas, video, audio, and graphics
- 09 December, 2010 01:27
The five characters HTML5 are now an established buzzword, found everywhere on the Web and often given top billing in slides, feature lists, and other places where terms du jour congregate. Nonprogrammers who must either manage or work with programmers are even beginning to pick up the term. Just two days ago, someone who can't manage a TV remote explained that he was sure his company's Web presence would be much better because they were using HTML5.
The five characters are in reality just the name of a document that isn't even finished. The W3C, whose job it is to build standard-setting descriptions of Web technology, has been contemplating the fifth version of the HTML standard for almost seven years. The latest HTML5 draft may finally become official in 2022, at least according to Ian Hickson, one of the authors who works for Google.
That is clearly too far in the future for many bosses and potential clients, who've put the HTML5 buzzword on their checklist. The good news is that most modern browsers have implemented a solid collection of the features, and it's quite possible to put HTML5 to work on your site for users are equipped with the latest browsers.
The biggest questions center on the newest technologies entering the standard. Many of these have been around for several years in various forms in various browsers, but now that the standard is coalescing, the major browsers are all lining up in support.
The new features can roughly be split into four major areas:
* Flashy new presentation tools. The pun is intended, as the new layers will make it easier for designers to create slicker graphical extravaganzas with a collection of tags. The <video> and <canvas> tags go a long way to replacing Adobe's Flash.
* Data layer. Web pages once required a connection to the Web. The new tools for handling data are turning Web pages from display mechanisms for distant servers into stand-alone software applications that store data without a Web connection.
* Other tidbits. A surprisingly large number of little ideas are bunching under the umbrella of HTML5, even if they're not in the official specification. New tools for using geolocation information and better organizing the presentation of data are all included.
* Little fixes to old approaches. The browsers have never handled every detail in uniform ways. The HTML5 standard cleans up some of the differences.
This article is the first of a series exploring how the browsers and developers are working toward HTML5, beginning with the new opportunities in the presentation layer.
Alas, artists don't think like computer programmers, and they don't grok the idea of separating the data layer from the presentation layer. When they want to put down a letter or a dab of color, they want it in one place and they want it to stay there -- or the harmony of the design will be forever destroyed.
The solution is the <canvas> element, an idea that originated with Apple, which incorporated <canvas> into the WebKit project. From there, it was copied by everyone but Microsoft. Eventually, Microsoft capitulated and added the option in IE9, but only after realizing everyone was using a free translation tool that mapped all of the methods from the <canvas> element into their own version.
To test the Canvas tags, I built a set of flying links that are animated using a jQuery plug-in written by Graham Breach. The code looks for a set of links in your HTML, then starts drawing them in a cloud that rotates in response to the mouse. It illustrates a few of the actions that can be achieved with the Canvas element.
The code won't work in IE8 and earlier versions of Internet Explorer, which don't support Canvas directly, but it will work if you use a thin translation layer called ExplorerCanvas. This code will turn Canvas calls into code that is understood by the earlier browsers, a technology Microsoft called VML (Vector Markup Lanaguage). This translation layer will be necessary for anyone who needs to support the older but still very common versions of IE.
Native <canvas> support in current Web browsers
Apple Safari 5.0 Yes
Google Chrome 8.0 Yes
Microsoft IE 8.0 / 9.0 No / Yes
Mozilla Firefox 3.6 Yes
Opera 10.60 Yes
HTML5 video and audio
The <video> tag looks perfectly innocuous, like an <img> tag where the pictures move. Anyone can add a video to a page by inserting the <video> tag, uploading the file to the Web server, and putting the right path in the source attribute for the tag. Voilà.
If only progress were that easy. While the format for the tag is always the same, the format for the file varies dramatically. This might not matter if the browsers all supported the same collection of formats as they do with the GIF, PNG, and JPG files used to hold still images. But they don't. Everyone has their own idea of which video formats belong and support can change from revision to revision of the browsers.
The most common formats are the Ogg Theora, developed by an open source group, Xiph.org; H.264, built by the MPEG group; and VP8, created by On2, a company recently purchased by Google. There are others, but they don't have as much support. Apple's Safari browser, for instance, displays anything that can be decoded by QuickTime.
All three of the main formats offer pretty comparable results, although video specialists can often find artifacts to argue about. The biggest point of contention is the cost of satisfying anyone who might hold a patent. The H.264 format was one of the most commonly used formats in the past, but it requires payment for patent licenses from the MPEG LA group. As an advantage, there are a number of chips that can speed up reconstructing the pixels from the compressed MPEG file. These can lower the battery consumption of handheld devices like the iPhone.
The On2 patents are now available for use royalty-free, which makes them more desirable for everyone. At this time, no one is publicly discussing patents that might govern the Ogg Theora format, but there are worries that the patents filed long ago may emerge some time in the future, giving the patent holder the right to sue everyone who's been using the format since then.
The patent questions may not be too important for small website designers and filmmakers because many tools come from companies that pay the licensing fees for encoding and decoding the H.264 stream. That license may not be enough for bigger fish, however, because the MPEG LA group also wants fees for broadcasting the video to big audiences. If you reach certain thresholds, you may need to buy this license or you may not. One MPEG LA press release ] promises not to charge patent royalties on rebroadcast before 2015. After that? It's all a bit complicated. The lawyers won't be cut out of the loop for some time.
Is there a solution? Not an easy one. While everyone continues to experiment with all of the features, the best solution is probably to encode the video in multiple formats, then use a script to detect the browser.
Native <video> support in current Web browsers
Apple Safari 5.0 Yes
Google Chrome 8.0 Yes
Microsoft IE 8.0 / 9.0 No / Yes
Mozilla Firefox 3.6 Yes
Opera 10.60 Yes
A more entertaining way of testing HTML5 video in your browser is found in this table filled with flying InfoWorld logos. If you can see the logo, your browser supports the format.
The audio tag is similar to the video tag, largely because it's just a subset. The <video> tag produces both sound and images, and the audio tag produces only the sound. In most cases, the audio tracks are a subset of the video format. The popular MP3 format for music and sound is a subset of the MPEG standard for encoding video.
For now, it seems as if the audio support will largely mirror the video support. If Firefox is pushing Ogg Theora video, then it also supports the Ogg Vorbis audio format because Ogg Vorbis is used for the audio portion of Ogg Theora. If some browser supports WebM for video, it will undoubtedly support WebM for audio. The one inconsistency may be the MP3 format, which is technically part of the general MPEG-4 video format but is now so common that browsers will support it independently. My latest version of Google Chrome, version 7, supports MP3 but not MPEG-4.
Video and audio formats supported in current browsers
H.264/MPEG-4 Ogg Theora / Ogg Vorbis VP8
Apple Safari 5.0 Yes No No
Google Chrome 8.0 Yes Yes Yes
Microsoft IE 8.0 / 9.0 No / Yes No / No No / No
Mozilla Firefox 3.6 / 4.0 No / No Yes / Yes No / Yes
Opera 10.60 No Yes Yes
Scalable Vector Graphics
As I mentioned in the Canvas section, the Web was built on the idea that data would be delivered in one format, then rendered or interpreted by the local computer. The Scalable Vector Graphics (SVG) format was invented to translate ordinary ASCII text into graphical shapes. The first committees began in the relatively ancient year of 1999, but the hard work is finally reaching fruition. Now that Microsoft is adding SVG support to IE9, all of the major browsers will support the format, more or less.
The goal of the SVG format is to bring the same kind of infinite precision to specifying a drawing that PostScript brought to print documents. Instead of rendering objects by specifying pixels, the drawing is made of lines and shapes, which are spelled out with tags like this one for creating a circle with a radius of 40 pixels:
<circle cx="100" cy="50" r="40" stroke="black"
The results are great for line drawings because the browser can create an image tuned to the resolution of the screen. The user can zoom in or out and the video card renders the result. Animated videos and games made of animated sprites are easy to specify and deliver because they can be so small. Even though all of the tags make the format insanely prolix, basic gzip compression can remove 80 percent of this syntactic lard. The SVGZ files are precompressed.
Does Microsoft's arrival at the party mean that everyone speaks the same language now? Of course not. Cell phone companies worried about wasting battery power on ornate SVG panoramas came up with two smaller subsets: SVGB (SVG Basic) and SVGT (SVG Tiny). These leave out a number of features that probably don't add much to tiny mobile screens, such as the ability to specify dimensions in values other than pixels. There are no picas or millimeters, and SVGT allows solid color fills only.
There are also numerous differences on the desktop implementations of the browsers. While they're all said to implement the core version of SVG, some developers are experimenting with doing even more with the flexible standard. One group, for example, started adding new <animate> tags that describe a path for objects to move along. After drawing the objects, the SVG engine can calculate a new location for them and draw them again. All the same math and tag infrastructure for drawing the lines can be reused to plot the path the objects should follow. At this point, the Gecko and WebKit browsers support the feature, but IE 9 hasn't built it yet.
Another plan is to let the browser apply all of the SVG effects such as filters and clipping to any arbitrary piece of HTML. The Mozilla team first implemented this and created a draft of SVG effects for CSS for the HTML5 team to consider. The other browsers offer some support, but they're widely considered to be more buggy -- and inconsistent with the way the ideas are implemented in the Mozilla browser. This shouldn't be surprising, given the same people at Mozilla wrote both the code and the spec.
A third area of inconsistency is in the implementation of SVG fonts. Unlike PostScript, the basic version of SVG doesn't make it easy to turn any arbitrary pattern of lines into a glyph. The WebKit browsers offer basic support for SVG fonts, so it's easier to add words to SVG drawings. IE and Gecko are still working on this.
Another potential extension has also been adopted by some browsers. After creating the rendering engines that produce infinitely crisp images with endless precision, some SVG users took a step back and decided the images might be nicer if they could be a bit blurrier. (Some people are never happy.) In any case, Chrome, Opera, and Firefox offer SVG filters that average all nearby pixels to produce smoother, prettier, often bizarre effects that aren't as cold or as crisp as line drawings.
My quick SVG test offers an easy way to check your browser's ability to host both embedded and framed SVG tags. A good collection of SVG examples and tests collected by Andre M. Winter shows many of the ways that people are experimenting with the basic SVG standard, as well as several of the proposed additions and extensions.
Native SVG support in current and future browsers
Displays SVG SVG effects for HTML SVG filters SVG fonts SVG in CSS backgrounds SVG animation
Safari 5.0 / 6.0 Yes / Yes Partial / Partial No / Yes Yes / Yes Yes / Yes Yes / Yes
Chrome 8.0 / 9.0 Yes / Yes Partial / Partial Partial / Partial Yes / Yes Yes / Yes Yes / Yes
IE 8.0 / 9.0 No / Yes No / Partial No / No No / No No / Yes No / No
Firefox 3.6 / 4.0 Yes / Yes Yes / Yes Yes / Yes No / No No / Yes No / Yes
Opera 10.60 / 11.0 Yes / Yes Partial / Partial Yes / Yes Yes / Yes Yes / Yes Yes / Yes
Source: When can I use…
WebGL: 3D Canvas
The WebGL format helps bring all of the power of OpenGL to the Canvas. Many of the development versions of the major browsers already support it, but they don't turn this on at the start. You need to activate it yourself by twiddling some of the configuration files. Chrome, for instance, asks you to start the browser from the command line and include the parameter --enable-webgl. Firefox lets you turn WebGL on from the about:config page (type "about:config" into the address bar).
The technology is not officially part of the HTML5 standard, but many people mention it in the same context because it is both experimental and tied to the Canvas object. The performance, of course, depends heavily on your video card and the memory available. Users with powerful graphics hardware report running an open source port of Quake 2 at 20 to 30 frames per second.
Andor Salga offers a collection of WebGL routines that test most corners of the spec.
More HTML5 goodness
The next several articles will dig deeper into the various enhancements to the HTML5 standard, many of which are hidden from view. While the <video> tags are hard to miss, the HTML5 standard also includes a number of ways that a website can begin to act like locally installed software. The second article will focus on local data storage, a set of features that lets a website stash information on the local disk and even access it when there's no Internet connection. The third in the series will dig into the newer ways that the new Web apps can communicate when there is a connection. Finally, the fourth piece will summarize a number of new features, such as geolocation, that don't fall into an easy pile.
All of these sets of new HTML5 features will radically expand the abilities of Web applications to be more than just a pile of links.