UI Design Standards: Any good in VR/AR?

In previous articles I've discussed general OS-level spatial design concepts, and touched on early visions of how you might present web content in that space. Today I'd like to get into some of the specific elements we use in user interface design (in apps, applications, and on the web) and how they translate into spatial interfaces, given the advantages and shortcomings thereof. Consider it a first foray into thinking about some of these topics rather than an authoritative dissertation.

General Considerations

The maturation of mobile UI design has actually been an excellent precursor to spatial UI design due to the imprecision of interaction models in both modalities. With mobile it's due to our fat fingers and the small screen size. With VR it's because our motions are being tracked less precisely and because we're mostly tracking large muscle group movements originating at our shoulders and neck. As we move to pupil and focal plane tracking and actual precise hand and finger tracking we'll do better. In addition some form of the HoloLens clicker will be useful. Overall, though, interface elements will be larger than has been traditional, and need to be more contextually adaptive as well.

Because it looks 3d and physical, people will react more like in the real world than like they do with a mouse and keyboard.

iOS 1 Chic

When Apple first released iOS it made many things look like physical objects were were already familiar with. Buttons and UI elements had leather or metal texture, the YouTube icon was an old TV set, and Notes was a nicely rendered yellow notepad. As people because used to iOS Apple started simplifying and abstracting to create a cleaner aesthetic. I think in spatial design we are in a similar place as iOS 1.0 was.

It's a whole new way of interacting with digital content, with a whole new interaction model, and we could do worse than to base some of our initial designs on physical things people are already used to. Because it looks 3d and physical, people will react more like the real world than like they do with a mouse and keyboard. You know how to push buttons, pull levels, turn knobs, etc. You don't really have drop down menus or scroll bars or arbitrarily delineated areas where a thing can exist. Your coffee mug doesn't disappear past a certain point on your desk and neither should your content.

Microinteractions

On the web and in mobile apps microinteractions are incredibly important. They're the little subtle animations that occur when you click on a button, a text entry field, or other page element. They give the user a clear sign that what they did was noted and in the best instances give them a clear indication of what to expect next. This importance is magnified in spatial interfaces. To push on something with your hand and have it not react, and in fact just have your hand pass through it, is disconcerting. Haptics are to be encouraged for those with motion controllers (like the HTC Vive or Oculus Touch), and some response animation is expected from anything you can interact with. These animations really can cause the much sought after delight in the user.

Main Drop Down Menu

The traditional drop down menu, used in most operating systems and applications for decades, is generally always there at the top of the screen offering access to various functions and shortcuts grouped by category.

These types of menus certainly have their place in spatial interfaces, at least in the short term. There aren't that many better ways to stow away a multitude of functions, which is why this type of menu has lasted this long. Of course there is no 'top' of your screen or window in spatial UI, so we'd have to either peg it to the top of your peripheral vision no matter which way you turn your head, or peg it to the top of the content itself, whatever form it takes. One of the benefits of spatial interfaces is that content doesn't need to be locked into boxes any longer.

Panels: Attached and Floating

In most traditional applications you have various menus that are grouped into rectangular panels. Sometimes these are in a set space within the UI and sometimes they can be arranged by the user: they float. In an application like Photoshop you have one for tools, one for text, one for layers, etc.

You can see how these panels can get incredibly disorganized if you aren't careful. When you add in a z-axis to the possible placement of menus it's even worse. Obviously you still need access to this multitude of tools in more complex software packages. In Photoshop you can use the TAB key to toggle the visibility of the menus.

If we think about menus floating 'above' the content, we could use the user's focal plane to bring them in and out of existence. So if the user focuses his eyes closer than the content layer the menus fade into view with contextually relevant ones as shortcuts in the center. This would be preferable to having the menus farther away since we'll likely have other content and applications farther away, not to mention the base level "desktop."

Your coffee mug doesn't disappear past a certain point on your desk and neither should your content.

Scroll Bars

Since content appears in rectangular windows we need scroll bars to move up/down and left/right. Quite often, particularly on the web, we end up with issues with content sizing differently on different resolution devices and we have scroll bars where we don't really need them.

With a mouse a scrollbar can be small and unobtrusive thanks to the precision of the instrument. In spatial we have to rethink the paradigm in my opinion. There has to be a standard for how we move content clusters (application windows in current OSes), move content within a cluster (like the actual excel chart within the window), and manipulate individual things within that content. It's a little weird having content hidden against an artificial edge of a window when dealing with spatial pseudo-physical interfaces. Perhaps a better model here would be that the content bends toward or away at the sides, almost like a microfiche machine. Whatever the method having it disappear in a clean line with no visual reason will be disruptive to the user's thought.

It doesn't quite work cognitively. Even so we need a way to manipulate said content. I have elsewhere suggested that moving content clusters could be done by having a manipulation 'bubble' area in front of the content. Your hand would change color when in this region to indicate this interaction mode.

Windows Scaling/Movement

Applications are currently shown in windows. Usually you can click and drag on the top of the window to move it or click along the edges and corners to resize.

I believe my bubble interaction model mentioned about and elsewhere is one decent proposal for this, at least for the moving part. For people using a clicker and focal plane tracking versus hand motion for the movement we could fade in a "transform" icon overlay when they focus in the manipulation region z-depth, click and it sticks to your vision's position and depth until you let go, just like with a mouse.

Video

A quick aside about playing regular video in VR: much like how some TVs project the color of the edge of your video onto the wall behind the TV, we could blur/fade a colored drop shadow version of the video playing behind it to blend it with the environment. It could be an effect similar to the new Apple music album art drop shadow in iOS 10.

Toggle Switches, Radio Buttons, and Check Boxes

In UI design we understand that a toggle is used to turn something on/off (usually a setting), a check box is used to select multiple things in a group, and a radio button is used to select a single thing from a group. Sometimes, however, we see check boxes used to indicate having read a disclaimer or some such thing.

The move to spatial does little to alter the basic functional importance of these features. What becomes more important is their scale and predictable behavior. The must be large enough to easily select. We can even use depth and animation to our advantage to make them jump off the page so to speak, allowing even easier interaction. Clicking on associated text next to the check boxes and radio buttons must select the button. A lot of web sites neglect this little detail currently and it's bad form now - in spatial it'll be fatal.

Carousels

Big banner graphics are standard on the web in 2016, some of them filling your entire screen with some graphic or video, text, and animation. They catch the eye, ideally tell the user something about your brand or content in an instant, and add a coolness to the whole site. In VR/AR your website is likely to take on more of a location, a place people enter into when interacting with your company or content. A print shop might create a scene with billboards and vinyl-covered trucks in some fun location. A jewelry shop might have a peaceful marble-covered garden complete with bubbling fountain and open display cases.

All this is to say that the carousel is not necessarily going to take on the form we're used to, but it can still have a place. It could be text there. Buildings could spell out a logo. There are endless possibilities, but the idea of a large ever-changing billboard is something that has a lot of history. Just ask Times Square. If it were me I might create something like the famous scene from Bladerunner.

Dropdown Menus

When you have more than a few options selecting or filtering the default is to look toward dropdown menus. They can be invaluable to quickly find out what the user wants, whether it's picking a font out of a list or choosing the type of transactions to query on your bank's website. Here's the thing though: if your list is so long that the user has to scroll within the drop list itself, you probably need to rethink your informational architecture to alleviate this issue. One possibility is to create category groupings, and another is to add another step in the filtering to funnel people toward their goal.

Spatially these types of lists probably need a rethink. I've seen some decent implementations (SimpleVR video player does a decent job of file/folder navigation), but I think perhaps taking a page from mobile UI might be smart here. iOS uses a wheel format for these lists, so maybe starting from that point would be good.

One thing to note is that if the user is required to select something out of a drop list this sort of wheel design would be good for spatial UI. There's no need to make the user click on a box to open the drop list when we know they have to do that. Showing it with a default value shows that they need to select something if they don't like the default. For optional drop lists it's advisable that they animate open rather than popping open, particularly since it's likewise advisable that they open both up and down from the central space and that they are reasonably large in size.

Sign Up and Other Forms

One of the biggest advances in UX in the past decade, in my opinion, has been the simplification of the sign up form. We've gone from pages and pages with dozens of fields where a site collects every bit of information about you it can, to just asking for a name or email and a password.

These are all what we in the industry call very good things, but what happens in VR where everyone wants to make things float about wantonly in a 3d space? Well, one important part of sign up forms is creating a sense of security in the user. So I'd say that if we are to break my rule regarding not using rectangles in spatial UI, now would be the time. Having a sign up form, or a payment form for that matter, in a solid, contained space that seems apart from everything else around it, and that animates into and out of existence in a way that indicates the information you entered is sealed away and perfectly secure.

Value Sliders

Value sliders are often used to control volume, choose a value within a predefined range, or sometimes to select a range within that range using two points on the slide. They are currently quite popular in VR for setting multisampling or antialiasing intensity. This is one instance where it just works in spatial interfaces. It's completely natural to grab and move the nodes around, which makes sense since it's based off a physical design like we all used to have on our car A/C control panel.

Physics and Animation

Good transitional animation takes into account the physics our brains expect to see. If something animates up and to the right, it should have a downward-facing arc to the motion, to give the object weight. If it moves in a straight line our brains complain that it isn't quite right. Important currently, and all the more so in spatial UI, where we are essentially dealing with a physical space, even if it's a virtual one. In AR it'll be incredibly important since we'll be interacting with real objects concurrently with the virtual ones, and if they behave obviously differently it'll cause issues.

Accordion Menus

Accordion menus let us pack several disparate but related content pieces into a small cluster, showing only one at a time. I'm generally not a huge fan of them, though they certainly have their place. If considering using them for VR or AR I would just encourage you to be careful about it, and consider some physical representation rather than pure abstract, be it folding paper, drawers, or something else.

Progress Bars

These let the user know where in some process they are currently. Sometimes it's a largely automated process like copying files, and sometimes it's a user-paced process like signing up for something over several pages, or filling out a survey. In spatial this will still be important, but we have more options for how to represent it. In addition to the obvious animations, like a filling glass or such, we could actually show the stages the entire time, the later ones farther away and moving closer as you move through the steps. This would really give the user a more clear concept of how much there is left to do.

Dialogue Boxes and Modals

Popup windows must be treated with care in spatial UI. They can't be too close to the user's face because it'll shock the person. They shouldn't just appear, they should animate in (if a person suddenly appeared in your room right now you'd jump out of your seat). They should also be attached to a world space rather than user screen space (ie; they shouldn't "stick" to the user's face).

I'm going to stop here for now. It's not an exhaustive list, although it might be a bit of an exhausting one. Once again, this is all in flux, it's all very new, and great thinkers and designers will evolve these modalities and invent new ones in the coming years. I hope this has given you some basic ideas of the concerns and considerations to take into account as you plan for the coming wave of spatial user interfaces. Although traditional UI and UX isn't going to go away anytime soon, this is an area rife with opportunity for innovation and growth. Spatial UI is here and is growing, and spatial UX will likewise blow up quite soon. It's up to those of us in these fields to get ahead of the wave and make sure experiences are as good as possible, designed from a user perspective, and bring in the user base needed to fund proper development in the space.