b-log – betriebsraum weblog

Software Development, Human-Computer Interaction, Projects…

Extremely Efficient Menu Selection: Marking Menus for the Flash Platform

December 11, 2009

This is a topic I wanted to write about since a while now: Marking menus and their implementation in ActionScript 3.0 for the Flash Platform. I had the chance of extensively studying this menu technique over the summer and developed a fully working solution for the Flash Platform, i.e. for use in Flash, Flex and AIR applications. Marking menus have proven advantages compared to other interaction techniques, but they are still underused in applications and standard UI component sets lack appropriate components (which might be part of the reason that they are relatively unknown). With the recent interest in gestural interaction and multi-touch applications (Flash Player 10.1!), the topic becomes even more relevant. One of the main aims of this article is to increase the awareness of one of the coolest menu forms around and highlight a few aspects in terms of technical implementation.

The article is a bit long, here’s the outline:
Some Background Information
What are Marking Menus?
Advantages of Marking Menus
Disadvantages of Marking Menus
Technical Implementation
The Source Code?
Final Remarks

Some Background Information
Marking menus were originally developed by Gordon Kurtenbach (currently director of research at Autodesk) and evaluated in his Ph.D. thesis, which was supervised by Bill Buxton (currently principal researcher at Microsoft). Surprisingly, that already happened in around 1993. Since then, this menu form has mainly been subject of various academic research studies, with most of them suggesting more sophisticated variants of the original design (more on that below). However, despite significant benefits, marking menus have been hardly used in commercial applications. The most prominent example of a successful integration is Autodesk’s (former Alias) high-end 3D software Maya and according to their user studies, they are very popular with expert users of Maya as they enable them to execute commands with high speed and pleasure-of-usage factor.

What are Marking Menus?
Marking menus are an advanced form of radial menus that display menu items in a circular fashion. In normal linear menus, items are aligned sequentially. In marking menus, each item is arranged around the menu centre and corresponds to the slice of a circle. In this respect, they are quite similar to pie menus which you might be familiar with. The distinctive feature of marking menus is their additional mode that enables scale-invariant gestural selection by drawing a stroke towards the target item without displaying the menu. That way, selections can be executed very quickly with ballistic movements. In fact, the usage of this “mark mode” is very similar to flick-style finger motions that have become quite natural on touch-based devices for navigation. When a mark is drawn, the system also leaves an “ink-trail” on the screen, which is displayed in an idealised way when the mark is confirmed. Marking menus can also be hierarchical and support “zig-zag” marks to select nested menu items. Sounds complicated? It isn’t. It’s really an intuitive and at the same time powerful way of selecting items.

AS3 based Marking Menu

AS3 based Marking Menu: (Top Left) Menu mode with different item renderers, (Bottom Left) Pull-Out item, (Top Right) Mark confirmation mode with trail of parent items, (Bottom Right) Mark mode displaying the idealised line and selected item

Another key feature is their capability to provide users with a smooth transition between the two modes: “Menu mode” is intended for novice users and displays the menu after a short delay when the mouse button is pressed. “Mark mode” is engaged when the user immediately strokes into the direction of the target item. In both modes, the same physical movement is used to select an item, so novice users automatically rehearse mark mode movements. Users are encouraged to explore the menu system in menu mode and can smoothly transition to the more efficient expert mode with increasing experience. In addition to menu mode and mark mode, there’s kind of an in-between mode called “mark confirmation mode” that allows users to complete an initial mark mode selection in menu mode. Finally, users can also abort selections and reselect (or “back-up”) the hierarchy in nested menus.

Advantages of Marking Menus
This is just a rough overview of some of the key advantages and some of the points touch more complex HCI-related topics that are only briefly mentioned. (Please also note that the terms marks/strokes/gestures are used interchangeably here).

The speed advantages of radial menus can be simply demonstrated with Fitts’ Law. Since marking menus pop up directly under the current cursor position and all items are equidistant from the centre, they can be approximately reached at the same speed + pointing is enhanced as the radial slices are of infinite size and thus can be more quickly and reliably selected. (Fitts’ law only applies to the physical movement of the pointing device. Other important factors, such as search time/decision time for item selection, must also be taken into account when evaluating the overall performance of menu systems. Decision time, for instance, can be modelled with the Hick-Hyman Law…)

marking Menus and Fitts' Law

Item distances in linear vs. radial menus, (Bottom) Fitts’ Law equation: MT = Movement time, D = target distance, W = target width, a and b = empirically determined constants

Disadvantages of Marking Menus
The most often mentioned disadvantage, namely the limitation of their menu size, is basically the only real known drawback of marking menus (or not – depending on how you look at it): They only allow a limited number of eight menu items per depth and a maximum viable menu size with acceptable error rates. This has led some researchers to come up with new menu forms based on the original design. Examples are control menus, flow menus, simple mark menus, zone and polygon menus, wave menus and flower menus. Most of these solutions address the issue of limited menu breadth and support for a larger number of items, while some of them provide a way to combine the selection and adjustment of values into the same interaction. However, they either have different problems associated with them or they are exotic to such an extent that wide-spread user adoption and acceptance may be hard to achieve.

After reading a few papers about those alternate designs, one could argue that some researchers have been quite particular about the depth/breadth limitation ;). Since it has been shown that classic marking menus can support four items per level up to four levels deep or eight items per level up to two levels deep, at least 64 items per menu can be reliably selected. Considering that marking menus are particularly well suited for contextual commands, this should be more than enough for most applications. The creation of unintuitive item categories due to the size limitation might also be not really an issue as it has been found that users gradually become familiar even with complex menu hierarchies.

Another drawback is that of displaying the menu near the screen edges (which is inherent in all popup menu systems) but the issue can be somewhat mitigated by different techniques. One solution is to display a special “pull-out” menu item to bring the menu back into the visible screen bounds (this should make sense when I post a demo video…).

Technical Implementation
The implementation of marking menus can get quite complex and as it turned out, even Maya’s implementation (which is not open, of course) is not without flaws. While coding a simple single-level menu is not a big deal, things get complex with hierarchical multi-level menus and scale-independent marks (which is an important property for fast selections). Calculating the inflection points, resolving mark ambiguities, and dealing with screen limits are then some of the issues to deal with. The amount of work is also increased since marking menus have two different modes (or three if you count mark confirmation mode) and should provide different item renderers, skinning, XML configuration etc. Besides, some other subtleties, such as delays and selection distances must be carefully implemented to avoid impairing usability. Overall, it could be said that a proper implementation is not a no-brainer but it is well worth the effort.

Mark recognizer

(Right side) The original stroke (grey line), articulation points (red line), and resolved, idealised line segments (green line with blue circles) are displayed. (Left side) The line segments and the selected nodes are displayed in DataGrid and Tree components.

The Source Code?
If you expect a downloadable zip here, I’m sorry to disappoint you – at least for now. The first reason is that the code was written under time pressure and needs some refactoring and speed optimisation before releasing it (btw, the version I created doesn’t depend on the Flex framework but implementing an additional Flex 4 component using the new layout features would definitely make sense). The second reason is that there might also be some patent issues which need to be clarified first. I contacted Gordon Kurtenbach from Autodesk a while ago who kindly offered help but unfortunately I was too busy to follow up at the time. I hope we’ll soon find a solution that allows developers and designers to use marking menus in their applications (also in commercial apps – which would be awesome to say the least!).

Final Remarks
All in all, I hope the article shows that marking menus are a great and well researched interaction technique that could be a valuable addition to the usual standard UI components. Given that in many RIAs menu selection is a fundamental task, providing users with efficient access to the application’s commands can be a nice way to increase usability. For example, if you think of the various online editors, marking menus could be smartly integrated for contextual actions in some areas. With two built-in modes, both novices and experts could access an intuitive and powerful selection tool. What’s more, marking menus support fancy gestural interaction and iPhone flick-style-selection, which can be cool for touch-based apps.
I’m going to post a demo video soon so that the points discussed here can be seen in action…and…let’s see if there might be some interesting things available on markingmenus.org in the near future ;)

Filed under: Flex/AS3, HCI, RIA

13 Responses to “Extremely Efficient Menu Selection: Marking Menus for the Flash Platform”

  1. andreitt says:

    Very interesting article. Thank you very much. I’m waiting for the demo video now to see it in action.

  2. Flo says:

    Yeah, thanks a lot for this interesting article. At the moment I am working on a very similiar topic for my master study. I try to analyze how we can use gestures to navigate faster and more effective through listbased menus. Marking Menus a very useful approach for this kind of stuff :-) Thx!

    Best regards,
    Flo

  3. [...] Check it out! Categories: Links Published: 12.18.09 :: No Comments » Leave a Comment Name [...]

  4. switcherdav says:

    Great demonstration, but how can you totaly disabled the fash default menu ?

    I mean “about flash player” and “parameters” entries

    Maybe you can’t, so that the reason why you won’t share du source code

    Of course, in a Air application, there is no problems, but in the flash player ….

  5. felix says:

    Hi,

    the flash default menu cannot be disabled via ActionScript. There is a JavaScript workaround to enable right-clicking in Flash apps but it’s quite a hack… You could, for example, trigger the marking menu by CTRL + left-clicking or just by left-clicking on certain areas/buttons in the user interface…and you may rest assured that I will release the source as soon as I have some time left to clean up the code, prepare some examples etc.

  6. [...] autour du pointeur, du doigt en l’occurence. Chez Hébiflux, je viens de croiser un article sur les “marking menus” et une vidéo qui montre l’intérêt de ce type de [...]

  7. switcherdav says:

    thanks for explications

    Hope you can share the source code soon

    Great work and merry christmas !

  8. Sharedtut says:

    Good post, thank you for sharing.

  9. [...] Window Control Marking Menu 2010-03-26 While it’s actually about a marking menu implementation in Flash, this gives a good introduction: Extremely Efficient Menu Selection: Marking Menus for the Flash Platform. [...]

  10. [...] d’abord lu cet article, puis celui-là, et enfin j’ai vu cette vidéo [...]

  11. MarkingMenuManiac says:

    6 months and counting…. is code coming?… :-) Or can you at least put your demo up so I can play with it? I’m working on somthing similar for a different platform, and would like to see how you handle inflection points (mouse changes direction) and resolving the ‘idealized stroke’.

    For the record: there are no patent issues: the papers and research you worked from pre-date any patents, and are thus unencumbered — unless your code is based on undiscloded and subsequently patented algorithms?

  12. elay says:

    Hi great Post

    I have a question, maybe you can give me a tip.

    I actually develop a Ipad App wich have the funktionallity of the Autodesk Marking Menue with the centermenu and the Line wich focus the Menuitem ( the Classik Marking Menu like Alias ).

    So my question is i am actually informed tha Autodesk have patented the Use of Marking menu. Do you think i can get a Problem when i use the Technik ( not the Design like in Autodesk Produkts ) of the Marking menue in my Ipad app ?
    Or is the Patent of the Marking menu actally free ?

    hope you can give me a Tip.

    thanks

Add a comment

Follow comments to this post by subscribing to the comment feed.