b-log – betriebsraum weblog

Software Development, Human-Computer Interaction, Projects…

Extremely Efficient Menu Selection: Marking Menus for the Flash Platform

December 11, 2009

Home » Extremely Efficient Menu Selection: Marking Menus for the Flash Platform

This is a topic I wanted to write about since a while now: Marking menus and their implementation in ActionScript 3.0 for the Flash Platform. I had the chance of extensively studying this menu technique over the summer and developed a fully working solution for the Flash Platform, i.e. for use in Flash, Flex and AIR applications. Marking menus have proven advantages compared to other interaction techniques, but they are still underused in applications and standard UI component sets lack appropriate components (which might be part of the reason that they are relatively unknown). With the recent interest in gestural interaction and multi-touch applications (Flash Player 10.1!), the topic becomes even more relevant. One of the main aims of this article is to increase the awareness of one of the coolest menu forms around and highlight a few aspects in terms of technical implementation.

The article is a bit long, here’s the outline:
Some Background Information
What are Marking Menus?
Advantages of Marking Menus
Disadvantages of Marking Menus
Technical Implementation
The Source Code?
Final Remarks

Some Background Information
Marking menus were originally developed by Gordon Kurtenbach (currently director of research at Autodesk) and evaluated in his Ph.D. thesis, which was supervised by Bill Buxton (currently principal researcher at Microsoft). Surprisingly, that already happened in around 1993. Since then, this menu form has mainly been subject of various academic research studies, with most of them suggesting more sophisticated variants of the original design (more on that below). However, despite significant benefits, marking menus have been hardly used in commercial applications. The most prominent example of a successful integration is Autodesk’s (former Alias) high-end 3D software Maya and according to their user studies, they are very popular with expert users of Maya as they enable them to execute commands with high speed and pleasure-of-usage factor.

What are Marking Menus?
Marking menus are an advanced form of radial menus that display menu items in a circular fashion. In normal linear menus, items are aligned sequentially. In marking menus, each item is arranged around the menu centre and corresponds to the slice of a circle. In this respect, they are quite similar to pie menus which you might be familiar with. The distinctive feature of marking menus is their additional mode that enables scale-invariant gestural selection by drawing a stroke towards the target item without displaying the menu. That way, selections can be executed very quickly with ballistic movements. In fact, the usage of this “mark mode” is very similar to flick-style finger motions that have become quite natural on touch-based devices for navigation. When a mark is drawn, the system also leaves an “ink-trail” on the screen, which is displayed in an idealised way when the mark is confirmed. Marking menus can also be hierarchical and support “zig-zag” marks to select nested menu items. Sounds complicated? It isn’t. It’s really an intuitive and at the same time powerful way of selecting items.

Another key feature is their capability to provide users with a smooth transition between the two modes: “Menu mode” is intended for novice users and displays the menu after a short delay when the mouse button is pressed. “Mark mode” is engaged when the user immediately strokes into the direction of the target item. In both modes, the same physical movement is used to select an item, so novice users automatically rehearse mark mode movements. Users are encouraged to explore the menu system in menu mode and can smoothly transition to the more efficient expert mode with increasing experience. In addition to menu mode and mark mode, there’s kind of an in-between mode called “mark confirmation mode” that allows users to complete an initial mark mode selection in menu mode. Finally, users can also abort selections and reselect (or “back-up”) the hierarchy in nested menus.

Advantages of Marking Menus
This is just a rough overview of some of the key advantages and some of the points touch more complex HCI-related topics that are only briefly mentioned. (Please also note that the terms marks/strokes/gestures are used interchangeably here).

• MMs are well researched
• MMs seamlessly integrate two modes: Menu mode for novice users, mark mode for expert users and a smooth transition between both modes. Novices automatically rehearse physical mark mode movements and can efficiently transition to expert behaviour.
• Mark mode is about 3.5 times faster than menu mode and up to 10 times faster than conventional linear menus. Even though selections usually only take milliseconds or seconds, for some applications this can result in significant time savings over a longer time period.
• Ballistic and casual “flick” movements enable expert users to trigger commands with high performance without being distracted by the visual representation of the full menu.
• Marks are scale-independent, i.e. they can be casually drawn at varying sizes and thus enable fast selections.
• MMs are self-revealing: A known problem with many gestural systems is the fact that they are not self-explanatory. MMs, however, are self-revealing since they display all commands in novice mode and show how they can be invoked. This is also an aid for expert users who wish to occasionally refresh their memory of available mark mode gestures.
• MMs better support cognitive skills: Command mappings of keyboard shortcuts are often inconsistent and users frequently fail to transition from menu selection to using the keyboard. Marking menus support stroke based command execution which has been shown to be more effective in terms of learning and recall. Marks may also be memorised in motor/muscle memory and compound marks can be “chunked” into a single movement.
• MMs can increase user satisfaction and the “pleasure-of-usage factor”
• MMs are well suited for mouse-, pen- and touch-based interaction
• MMs support “eyes-free” selection and are therefore accessible for blind people

The speed advantages of radial menus can be simply demonstrated with Fitts’ Law. Since marking menus pop up directly under the current cursor position and all items are equidistant from the centre, they can be approximately reached at the same speed + pointing is enhanced as the radial slices are of infinite size and thus can be more quickly and reliably selected. (Fitts’ law only applies to the physical movement of the pointing device. Other important factors, such as search time/decision time for item selection, must also be taken into account when evaluating the overall performance of menu systems. Decision time, for instance, can be modelled with the Hick-Hyman Law…)

Disadvantages of Marking Menus
The most often mentioned disadvantage, namely the limitation of their menu size, is basically the only real known drawback of marking menus (or not – depending on how you look at it): They only allow a limited number of eight menu items per depth and a maximum viable menu size with acceptable error rates. This has led some researchers to come up with new menu forms based on the original design. Examples are control menus, flow menus, simple mark menus, zone and polygon menus, wave menus and flower menus. Most of these solutions address the issue of limited menu breadth and support for a larger number of items, while some of them provide a way to combine the selection and adjustment of values into the same interaction. However, they either have different problems associated with them or they are exotic to such an extent that wide-spread user adoption and acceptance may be hard to achieve.

After reading a few papers about those alternate designs, one could argue that some researchers have been quite particular about the depth/breadth limitation ;). Since it has been shown that classic marking menus can support four items per level up to four levels deep or eight items per level up to two levels deep, at least 64 items per menu can be reliably selected. Considering that marking menus are particularly well suited for contextual commands, this should be more than enough for most applications. The creation of unintuitive item categories due to the size limitation might also be not really an issue as it has been found that users gradually become familiar even with complex menu hierarchies.

Another drawback is that of displaying the menu near the screen edges (which is inherent in all popup menu systems) but the issue can be somewhat mitigated by different techniques. One solution is to display a special “pull-out” menu item to bring the menu back into the visible screen bounds (this should make sense when I post a demo video…).

Technical Implementation
The implementation of marking menus can get quite complex and as it turned out, even Maya’s implementation (which is not open, of course) is not without flaws. While coding a simple single-level menu is not a big deal, things get complex with hierarchical multi-level menus and scale-independent marks (which is an important property for fast selections). Calculating the inflection points, resolving mark ambiguities, and dealing with screen limits are then some of the issues to deal with. The amount of work is also increased since marking menus have two different modes (or three if you count mark confirmation mode) and should provide different item renderers, skinning, XML configuration etc. Besides, some other subtleties, such as delays and selection distances must be carefully implemented to avoid impairing usability. Overall, it could be said that a proper implementation is not a no-brainer but it is well worth the effort.

The Source Code?
If you expect a downloadable zip here, I’m sorry to disappoint you – at least for now. The first reason is that the code was written under time pressure and needs some refactoring and speed optimisation before releasing it (btw, the version I created doesn’t depend on the Flex framework but implementing an additional Flex 4 component using the new layout features would definitely make sense). The second reason is that there might also be some patent issues which need to be clarified first. I contacted Gordon Kurtenbach from Autodesk a while ago who kindly offered help but unfortunately I was too busy to follow up at the time. I hope we’ll soon find a solution that allows developers and designers to use marking menus in their applications (also in commercial apps – which would be awesome to say the least!).

Final Remarks
All in all, I hope the article shows that marking menus are a great and well researched interaction technique that could be a valuable addition to the usual standard UI components. Given that in many RIAs menu selection is a fundamental task, providing users with efficient access to the application’s commands can be a nice way to increase usability. For example, if you think of the various online editors, marking menus could be smartly integrated for contextual actions in some areas. With two built-in modes, both novices and experts could access an intuitive and powerful selection tool. What’s more, marking menus support fancy gestural interaction and iPhone flick-style-selection, which can be cool for touch-based apps.
I’m going to post a demo video soon so that the points discussed here can be seen in action…and…let’s see if there might be some interesting things available on markingmenus.org in the near future ;)

FON theme by Leon Paternoster. Powered by WordPress.