Alex Faaborg

Previously
Part 0: Introduction to Microformats
Part 1: Structured Data Chaos
Part 2: The Fundamental Types
Part 3: Introducing Operator

Part 4: The User Interface of Microformat Detection

Now that I’ve posted about what microformats are, why the browser should natively detect them, which microformats should be detected, and Mozilla Labs has introduced Operator, it’s time to take a look at how microformat detection could potentially appear in the user interface of a future version of Firefox.

20 conceptual mockups of Gran Paradiso and a wealth of UI discussion and debate after the break…

Technorati Tags: Firefox 3, Gran Paradiso, microformats

Please note that all of these images are only conceptual mockups of Gran Paradiso (the codename for Firefox 3). The final design may be entirely different, or the idea may not be implemented at all. This point is particularly salient since some of the ideas in this post are likely to be controversial. The purpose of these mockups is to explore a variety of different ideas and generate feedback. However, if you are really curious about what Gran Paradiso looks like, some actual images of Gran Paradiso are available here. (note: that photo is geotagged, if you have Operator, check out the satellite view with your favorite mapping application).

This post is pretty long, so if you are short on time you may want to just scroll down and look at the mockups. On most of them you can click through for the full image. Inspiration for many of these designs came from the microformat community.

Integrating Microformat Detection UI into the Browser Chrome

Right now Operator appears as a toolbar, or (with version 0.6.1 and later) in the status bar. There are a number of other places in the chrome that could potentially be used to display the detection of microformatted content:

Location Bar

For instance, a icon could appear in the location bar next to the feed icon:

I’ve previously stated that I really don’t want to let the right side of the location bar spiral down into Firefox’s equivalent of the Windows system tray (which after getting a new computer from an OEM is usually inundated with meaningless little device driver icons that blink and flash at you without reason or consequence). At the very least we could integrate all of the detected microformats into a single plus icon to reduce the overall visual complexity:

Note that similar to the design of Operator, microformats are each displayed using the icon of the associated application. In Firefox mockups for Vista, microformatted content is shown paired with a mixture of Microsoft and Google apps. In the mockups for OS X, geo and adr happen to be associated with Google Earth, hCard with Address Book, and hCalendar with iCal, (which are just the apps I personally use). These are just examples, the application associations will be up to the user.

All of these mockups display application icons instead of the generic microformat icons shown in part 2. This is because application icons are more visually unique, they are personalized to each user, and they hold a strong connotation of what will happen when you click on them.

In addition to the fundamental types of microformatted content that are natively supported, these mockups also display microformat detectors added by the user. In this case the user has added Skype, but they could have added something much more domain specific, like an extension that detects human genes, or wines. Some of these third party extensions will detect specific microformats, others could potentially use regular expressions, or detection lists to locate data and associate it with an action. It is important to note that even if we support detection based on regular expressions or detection lists, these would probably all be user-installed extensions. Integrating detection into a single UI is more consistent and usable than than having different interfaces for each data type.

Sidebars

The Content Sidebar
One way to expose microformated content in the browser chrome is with a sidebar:

The idea of a Content sidebar could also improve the user experience of batch downloading media (images, movies, etc.) from the Web.

Sub-Page History
Microformated content that users encounter on the Web should also be stored in history. This information should be searchable, and users should be able to browse their history for particular types of information. There are a variety of cases where this might be useful. For instance, imagine you are searching for apartments across a variety of different sites, and then at the end of the browsing session, you want to send everything you saw to Google Earth:

This mockup also displays an idea for redesigning history that is described in a separate post.

Sub-Page Favorites
What if you could use the same tagging scheme for bookmarks, and saved pages, as well as people, locations and events you encounter on the Web? In this conceptual mockup I have tagged some Web pages related to my upcoming recruiting trip to MIT, but I have also tagged events (my flight to Boston, the career fair), locations (my hotel, the career fair), and people (who I am going to meet while I am there). I am able to very quickly retrieve this data because all of it has been tagged with “MIT recruiting 2007.” In this mockup I am only using Google apps.

The Need for Microformat Detection, In Context

Here are some different ways you may wish to interact with microformatted content you encounter while browsing the Web:

Acting on one item out of a small number of items
Let’s imagine you are reading your friend’s blog post about a concert that is coming up, and you want to add the event to your calendar. Since there aren’t any other events on the page, or only a few other events, you can easily send the event to your calendar using a menu based interface like the one Operator provides.

Acting on all items
You are looking at your credit card statement and are wondering what a few of the charges could be. You send the 117 events to a new layer in your calendar so you can see the bank statement overlaying what you were up to at the time. Again, this is easy to accomplish with a menu interface, it just needs a Send All command. (Getting banks to update their statements for microformats is of course a separate challenge.)

Acting on one item out of a large number of items
You are viewing your company’s online phonebook with 80 entries and want to send a particular coworker’s contact information to your cell phone so you can call her when you are away from your computer. This is difficult with a menu interface because even after you locate the information on the Web page itself, you need to find the same entry in a menu that contains 80 items. It would be considerably easier if you could search the Web page, then click directly on the phonebook entry to send it to your phone. You really need to be able to act on the information in the context of the page.

Acting on a selection of items
You are browsing a Web site for a new apartment, and of the 50 search results returned, 15 of the apartments look like places you could see yourself living. You want to select the 15 apartments and send them to Google Earth to compare their locations. What you need for this scenario is not just the ability to act on information on the page itself, but to command/control-click items to select a subset, and then take action on that subset.

Because of the last two interactions, it would be very useful to display detected microformatted content, in context, on the Web page. However, a Web browser modifying the content area is a little controversial.

Learning From Past Mistakes

With the exception of highlighting search terms in the page, Web browser UI generally doesn’t extend into the content area, except for one time, awhile ago. Before the release of Windows XP, Microsoft was planning to integrate Smart Tags, a feature previously in Office, into Internet Explorer 6. Of course there are several considerable differences between Smart Tags and microformat detection, most notably that content creators are personally adding microformated content to their sites because they want it to be detected. However, from the IE6 Smart Tags fallout, we can learn several lessons about how to more respectfully handle the user experience:

1. Don’t use color
The Web is a big place, whatever color you choose (including purple) is bound to clash.

2. Don’t leverage an existing monopoly in an effort to redirect users solely to properties that are controlled by your particular corporation
Users deserve choice, and application developers deserve an open and extensible platform

3. Don’t take control of the Web page’s presentation away from the Web designer
Web designers should always have total control of what their site looks like. They should be able to stop the browser from overlaying UI on top of their page, or decide to customize the overlays so that they will match the page’s visual design.

The nature of microformatted content creates some additional design constraints:

4. Hyperlinks won’t work very well
Hyperlinks are designed for navigation on words, or short phrases, not for large sections of text. Microformats (locations, events, and contacts), will each contain several pieces of information, and the pieces themselves may already be hyperlinks.

5. Single icons won’t work very well either
Adding single icons () to a Web page does not work well because in many cases it is not always clear exactly what information on the page the icon maps to. To avoid ambiguity, the presentation of microformated content needs to rely on area, and not proximity, in terms of the gestalt principles of grouping (PDF link, see page 25 if you are curious). This is also important so that microformat detection appears to be overlaid on top of the Web page, as opposed to injected into the Web page.

Taking these five considerations into account, here are a some conceptual mockups of ideas we’ve been thinking about over the last several months.

So you can note what is being added, here is the page without any overlaid UI.

Modal Designs

First, I proposed the idea of rendering transparent panes of glass over areas of the page that contain microformated content in a mozilla.dev.apps.firefox thread back in August. Note that in these modal designs, the overlays only appear after the user has requested to see microformatted content in the page. In this case by selecting one of the items in the toolbar, or . Here is what the panes of glass idea might look like:

This design technically violates the first rule, because the shade of grey created by the transparent pane of glass is a color. Mike Beltzner had the idea that on dark pages these panes of glass could actually produce a light source:

Mike Shaver also proposed shining spotlights on the page, similar to the search in OS X’s system preferences:

If this looks too fuzzy, we could black out the page and then outline the microformatted areas:

Another slightly more wacky idea I had was to literally push the page towards the user in areas that contained microformated content. Ideally, this design would be implemented with graphics acceleration (which may potentially be possible with Cairo). It’s hard to get the whole idea from a static mockup, but imagine the raised sections have blocks behind them that push towards you when you ask the browser to display microformatted content:

Since the user will be launching an application (either online or locally) when sending it microformated content, perhaps we should have a design that is reminiscent of each platform’s UI for application launching. Here is a design that contains visual elements of the dock for OS X Aqua:

And the same design on a black Web page:

One nice thing about the dock is that it is designed to provide strong visual cues regardless of the background image it is superimposed over. In case you are curious how the back of the dock is implemented: the center of the dock is white with a 40% opacity, followed by a single pixel border of white with 50% opacity, followed by a single pixel border of black with 13.5%(ish) opacity.

This idea doesn’t really carry over as well to Windows. Here is a design that uses visual elements from the start menu to match Vista’s Aero:

Another approach would be to match the appearance of each platform’s UI for file selection. Here is what that could look like on OS X:

And Vista (violating the rule about color to be consistent with the user interface of managing files):

Taking considerably more creative freedom (and this time really violating the rule about color), here is a design created to compliment Ubuntu’s Human theme. This design could have an interesting appearance animation if the outer parts expanded while the center faded in:

In these modal designs, clicking outside of the highlighted areas will return the page to it’s normal display, like the Dashboard in OS X. This is referred to as a spring-loaded mode. Also, all of these designs support multiple selection (with some visual indication that an item is selected), and copy and paste (through a contextual menu, the edit menu, and keyboard shortcuts).

An issue that these mockups don’t resolve is what happens if the user attempts to click on a hyperlink that is under an overlay. We would need to treat the action as selecting the item, but this will result in mode errors. This issue, in addition to others, is why we probably want to go with a non-modal design.

Non-Modal Designs

The modal interfaces described above do not proactively modify the Web page, the user has to actively ask for the microformatted content to be displayed. This UI has the advantage of getting out of your way when you are not interested in acting on pieces of content, but the disadvantage of not being discoverable. When microformats first start to appear on the Web, people may regularly invoke the microformat display mode just to see if they can perform an action on a particular piece of data, or they may forget about the feature entirely. Since neither of these situations is ideal, another approach is to always display application icons next to microformated content, but in a less obtrusive manner. For instance, just the icons would appear on the page, and then an overlay indicating the outline of the microformated content would appear when the user hovers over the icons:

The user hovers their mouse over the icons and the entire area is highlighted:

All of the above designs could be adapted to a non-modal interaction (although blacking out the page on hover might be annoying, so we would likely want to avoid those designs). Initially I preferred the modal design, but Mike Beltzner successfully convinced me last week that non-modal is a better solution. Note that even with a non-modal display the user could still turn the overlaid UI on and off (and we are back to modal), so the issue is really if microformatted content should be displayed by default, and how intrusive the display should be. Which type of interaction style do you prefer?

Acting on Content

In all of the designs above the icons for each associated application were displayed. This strikes me as the most simplistic and intuitive approach. However, there are a few other things we could do:

Descriptive Text
This design simplifies the initial display, and provides the user with textual descriptions of the available actions.

Pie Menus
For these types of contextual actions, pie menus provide a great UI because they are are faster and more reliable than linear menus. Another advantage of pie menus is that over time you will learn which direction particular actions are. This makes interacting with them essentially a small mouse gesture, and you don’t even have to focus directly on the menu to hit the right target, you can be viewing the information you are acting on.

Pie menus have been around for awhile now, they date back to at least 1988, and in terms of the Mozilla project, back to 2002. In the past, pie menus have been a popular UI control for contextual actions in video games.

Why Should Firefox Display Microformatted Content in the Page?

I realize that extending the browser UI into the content area is a controversial decision, and whatever default graphics we overlay on top of the page have to be respectful to Web designers. Also, it is not clear if any of the above designs achieve that, since it is difficult to overlay user interface without modifying the appearance of the Web page.

However, here are some additional reasons why I believe Firefox should display microformats in context:

Personalization
If Firefox takes care of the user interface for in-page microformat detection, we can display the icons for the specific applications the user has associated with different types of data. We can’t expose that information to the Web site without violating the user’s privacy.

Consistency (of the interaction, not the appearance)
If Firefox takes care of the user interface for acting on microformatted content, then the interaction model will be the same across all Web sites. Providing a consistent interaction model will improve usability. Note that the actual appearance should be easily customizable, as described below.

Low Barrier to Entry for Web Developers
Adding microformat support to a Web site needs to be as easy as we can possibly make it for Web developers, otherwise no one will bother to use microformats. Sites on the bleeding edge of the technology adoption curve like Upcoming.org, Flickr, Eventful and Ma.gnolia are going to integrate microformat support into their sites (all of those already have), but what about your local bank that last updated its Web site in 2001? What about someone who quickly wants to put microformated content on a blog or wiki? (This is the topic of my next post on microformats). If people only have to add a couple of divs and spans to support microformat detection, that is an order of magnitude easier than asking them to provide the entire user experience. Supporting microformat detection needs to be easy enough that all Web developers need to do is just write a couple of lines of code in between a database and Web site to generate the additional semantic markup. Otherwise, most sites are not going to go to the effort of supporting microformats, and users will not be able to take advantage of using their favorite applications for particular types of data.

Compatibility
Another issue to take into consideration is: what if you are viewing a site that contains microformatted content in a Web browser that doesn’t support microformat detection? Of course the Web developer can detect what browser the user has and adjust the site accordingly, but that is just additional work. However, if the source just contains a few extra divs and spans, microformat enabled sites are completely backwards compatible.

Security
The final reason I believe Firefox should provide the user interface for microformat detection is security. If we allow Web sites to initiate the action of sending data to particular applications on the user’s computer, we expose all of these applications (instead of just Firefox) to attack. Plus, Web sites could attempt to passively send information, which means we would need to popup a dialog box to make sure the user actually wanted the transfer to take place, and we all know how those can both get annoying and condition the user to always click yes.

Giving Web Designers Control

While Firefox needs to provide a good default interface, we will of course allow Web designers to easily skin the detection so that it can match the look at feel of their site using CSS, or disable it completely. To allow some customization with less work, perhaps we could also ship Firefox with a variety of simple appearances for microformat detection, and allow Web designers to specify which they would like used for their site. In addition to selecting a design, perhaps they would also customize a single aspect, like the color. There would of course have to be a few checks for security. For instance, we couldn’t allow hit areas that cover the entire content area and are completely transparent.

If we ship with multiple styles of microformat detection, we may also want to let the user choose which one they want to use in preferences. It would also be great if the detection interface was extendable and contributers could create new designs, similar to how you can currently develop new themes.

Overall, I believe that regardless of what design we eventually end up going with as the default (if we decide to implement this feature in Gran Paradiso), it is important for the Web browser to provide users with microformat detection in the context of the page. Detecting microformatted content on the Web is going to significantly streamline the way we explore, share, archive, and interact with information.

A New Version of Operator Has Been Released

Can’t wait to try these features out? Operator 0.6.2, which was released recently by Michael Kaply, includes in-page microformat detection.

Comment

Should Web browsers go ahead and render UI over a Web page, if it makes life easier for users? Modal? Non-Modal? Location Bar? Toolbar? Sidebar? Pie! Which designs do you like? Which designs do you hate? Please voice your opinion below.

Contribute

Open source isn’t just about contributing code, any UI designers out there are welcome to contribute mockups. Please add your ideas and mockups (at any level of fidelity) to this page on the Mozilla Wiki.