15 Initial thoughts on best practice guidelines for Spatial user interfaces for VR and AR.
Making the content reactive to the behavior of the user is a key benefit of eye tracking and HMDs
As VR gains steam and high fidelity AR is waiting in the wings, we UI, UX, and Interaction designers need to get a leg up on how to go about it. In this article I intend to lay out basic OS and content guidelines for spatial user interfaces. I am assuming that eye tracking will be playing a key role in the next wave of HMDs, like the Fove and others. I am including some very basic animations demonstrating a couple of the concepts. In addition I am including a rudimentary chart showing how various content types relate to and other types of content.
1. No Straight Conversions, No Rectangles: The rectangular content and UI designs currently in use were created due to the shape of monitors and books. Just bringing these rigid designs into 3d space adds nothing to our ability to give the user superior access to our content, and does not open the door to new content types and methods of presentation. Some content will lend itself to rectangles, but it should not be the default.
2. Reactive to User Behavior: Making the content reactive to the behavior of the user is a key benefit of eye tracking and HMDs. Whether it’s knowing when to bring in an image or video or 3d experience based on the part of the article the user is reading, or popping up help when a user stares longer than normal at a certain image or piece of content, utilizing the gaze and user actions is something that needs to be designed for. In the crude animation below I show a way this could work. Note that in an HMD's field of view only the smallish area in the center is really usable for primary content, which is why the text area looks small in the video.
3. Simple backgrounds for content display: Most current 3d UIs have a cool environment and rectangular windows with solid backgrounds to display content in. A suggestion is to have plain backdrops for most content types so that you’re not restricted in the shapes and manners in which you display the various content pieces. People who design really clean web pages suddenly make things cluttered when they move to 3d. There's no need. Look to the new game Inside for a nice clean, but 3d, aesthetic.
4. Consistent placement of content pieces: One of the advantages of spatial UI is there’s more ‘space’ to put things. With this in mind it can easily become as chaotic as a messy desk, with the added problem of things appearing and disappearing constantly. So a keeping a conceptual persistence to positioning of similar information is incredibly important. One idea with informational content is to have the title move to the left and stay put while the user is reading/viewing the content in the center of their view. They can always refer back to the title on the left. In this way the titles of items the user has open can take on a ‘card stack’ style. Switching to another card will replace the central content with that card’s associated content. I show this (crudely) below. When the 'cards' come to the center that represents the user turning his head. Head turns would change your content type and eye tracking would handle specifics within content.
5. Areas of Influence: The user needs an area of space in which hands and arms and fingers can be maneuvered without fear of accidentally interacting with things. We can depict this by overlaying color onto the hands or hand models, changing to another color as they pass through an invisible barrier into an objects area of influence. The behaviors of gestures and such would then be dictated by the type of content the area surrounds. For example, if the content is a paragraph of text, touching a place in the text with a finger and dragging down/right could select the text between the two points.
6. Don’t Make Everything Neon/Translucent: Everyone wants to make stuff look like Tron, with translucency and glowy bits. Stop. Stick to the basics of good design: text should be easy to read (large enough, enough contrast), layers of content should make sense (card stacks, etc.) and not just be randomly positioned, things should behave predictably.
7. Use 3D where it helps user understand content: Not everything has to include 3d just because we can display it now. If the content is easier to see or understand presented as a 3d model or simulation, then by all means go for it. If, however, the content doesn’t relate to spatiality, then stick to basics lest it become a gimmick.
8. 508/Accessibility: The same principles of 508 apply to spatial UI, which means some of the above points are even more important. Text must be readable easily with enough size and contrast. Certain color combinations have to be avoided, which means too many environmental backdrops will be a no-no. Narration will be important, and offers and opportunity for companies to have a voiced avatar describe the environment. For example, a Tiffany’s webfront can include the ambient sounds of the NYC store, the murmur of people, and a narration by the welcoming host.
9. Sense of Place = Sense of Purpose: The sense of presence that HMDs convey can be utilized intelligently to prepare the user to perform certain tasks. A pleasant ambiance in a Tiffany’s store prepares you to make an elegant purchase. During checkout, however, that might seem too open. Enclosing the user both visually and aurally in a small office with opaque walls creates the sensation of security during checkout. Viewing a photo gallery might work fine with the French Alps surrounding the user but to get some work done simple backgrounds and more controlled arrangement of interface elements becomes important.
10. Ads Should Not Stick: The emotional impact of HMDs is much deeper and more visceral than with screens. Ad providers risk created a deep-seated resentment in users if the ads are too much in your personal space. The worst of this would be if ads “stuck” to the HMD, so that they stayed in place no matter how you turned your head or body. You should ALWAYS be able to walk or turn away from content, particularly advertisements, just like in the real world.
11. Common OS UI Should Stick: No one wants to have to turn their head to switch applications constantly. Certain standards, like the taskbar, shouldn’t be hidden away outside your periphery. Glancing down should let you access it. An experimental concept for multitasking would be to let the user lean forward and back to see various stacked content areas. This would also work their abs. Another idea would be to let the users position largish icons of the programs parallel to the desk/ground surface and select them with one’s hand.
12. Closing content: There’s no reason we can’t close content by keeping the “X” in the top right and letting the user look at it and click a mouse or other device, or touch with their hand tracking. If the application is using card stacks for tabs or multitasking we could use a standard swipe up to clear content pieces. If we have our open applications on our ‘desk’ we could close those by swiping them off to the side past as certain point.
13. True vertical text: One of the big issues with HMDs currently is the warping and perspective distortion that occurs when objects are displayed in dimensional space. Near-vertical lines are the worst due to their tendency to alias badly. Since nothing is true up-down for long in HMDs, this problem afflicts text more than anything else, particularly since text is such a vital part of both informational transmission and visual design. One potential fix is to have important body text be presented flat as an overlay on the 3d environment, mostly for lengthy paragraphs and such. Head turns would simply move the effectively orthographic view of the text around the text area. So tilting your head down will essentially move the text “paper” up. Even when HMD displays are 8K and higher resolution this will still matter.
14. Scary avatars: If you can’t have a 3d avatar that gets past the “uncanny valley” then you’re better off either not having one, having a disembodied metaphoric one (like a glowy ball or something), or having a cartoony one. This will come into play with social VR/AR experiences.
15. Know When User Has Ignored Content: Say you have content. The user is reading along and you pop up an image related to the line currently being read. If the user looks down at it and taps or clicks, the image will come closer to the user (effectively making it bigger and blurring the text due to eye focus). We must know, however, when the user has decided against opening that image. We can base this on when they start reading the following sentence, or when they scan back to the left margin of the text, or such. The exact methods will have to develop in response to user behavior, but it’s something to keep in mind.
So those last three are more conceptual, but still interesting. Allow me to present a simply chart for your perusal. This chart relates various content types to a particular type. So in one section everything is related to text. In another everything is related to an image. I've tried to show how they support and enhance each other, so that we can start from basics when designing content for this new modality.
You'll want to open that in a new tab if it's too small. Let me know if you have any questions. I think the list addresses many of the major issues I've seen popping up in current efforts. I'll have much more to come on this subject over the summer. Until then, best of luck!