360° Room-Scale Projections

too long;didn't read

We extended our image room projection to spherical images to cover the entire room. You can try it here corner.builders. See at the bottom of the article for a short demo video!

Where we are

As always, i recommend reading the previous newsletter editions as this is a work in progress, that builds upon the previous editions! As a reminder of "where we are": we can now use the website corner.builders to project a colorful image of a star in a room of custom dimensions, such that the image appears undistorted from the given point of view, then extract a list of images to print and install on the walls, see this video on the output I did from my son's bed point of view.

Next Goal: full 360° room scale projection

Right now we have a rectangular image floating in front of us that we project to the walls behind. But what if instead of this flat rectangle we had an image on a sphere all around us that we project back on all directions? This way we could have full room experience where we can turn our head around and look everywhere! Ok thats sounds cool but also like a substantial change.

So what core things do we need to change?

Required core changes:...nothing

We are already projecting from a 3d object: a flat mesh grid, but still floating in front of the viewpoint. Now this 3d planar grid will take the form of a 3d sphere around the viewpoint but the projection process remain the same: for each vertex in the mesh, compute the new position by drawing a straight line from the viewpoint to the first wall surface intersection. So really nothing much to add, beside modifying the original flat mesh !

360 image specs

We can choose from several 360 format. For now we will chose 1, but maybe later we can integrate more.

Skyboxes

Imaging a cardboard box that you unfold flat on the ground: it would look like this:

So any image that follows this layout can be mapped back to a 3d cube with the observer inside in the middle.

Equirectangular image

This format might be more familiar as it is commonly use to represent planet earth on a flat map. Each pixel row maps linearly from [-90°..90°] and each column maps from [-180°..180°] (latitude and longitude). Here is the same image as above, but with this specification:

Our choice and first try

For the sake of simplicity we will chose the equirectangular format. So we just need to map our previously [x,y] flat vertices to [longitude,latitude]. Supporting skyboxes is not particularly more difficult though, so we might add it later if needed! Below is an image of the outside-of-viewpoint looking at the floating sphere image, before projection!

yep, looks about right! a floating sphere with some colors, now lets adapt the GUI and look inside!

GUI exposition

Switch from Flat to 360

We need a way in our GUI to switch from "flat image" to "equirectangular image". A future feature could be automatic detection of the image type based on some AI classification. For now, for the sake of speed and keeping MVP in mind, we will have to select the image format. Simple radio-buttons should be enough.

Point of view definition and Horizontal horizon

What is the same and what is different for the point of view definition compared to the flat image case? We still need the 3d position (the head position). However we don't need the "scale factor" (making the image bigger or smaller by scrolling, it takes the whole space anyways).

Something that we need to restrict for the image orientation is the "pitch" of the image. Most 360 image are made "horizontal" meaning the horizon line exists and should be preserved. If we don't restrict it from our flat use case then the horizon plane will be tilted depending on how well the use manage to keep it horizontal. We need then to enable a setting by default that will keep the image horizontal plane, truly horizontal regardless of the camera pitch during the viewpoint setting phase. A first draft of the UI for the image could look like this:

We added a "Image" menu where we can select either Rectangular or Spherical. And if Spherical is enable, a new checkbox pop-up (already checked) that will keep the horizon plane"flat". (Next there will be an additional button to select the image from the device, coming soon!)

Room edges

As you saw in the previous capture, the little floating image sphere is way smaller than the room. If we are inside, then we don't see the room edges, and it becomes impossible to track the room orientation... Well, that's easy to solve, just always draw the room edges on top, regardless of the relatives depth!

The image choice

360 images can be captured from special devices. They can also be generated in a simulated environment to get more fantasy ones. The image generation itself is not the purpose of the tool developed here. So i'll take for default a nice one that I found from claudius-22 on flickr. The flattened image looks like this:

Testing time!

Ok, below is how this now looks in app!

Next steps

Non-dev: Actually printing and trying how it turns out for a small room like a small unused WC room I have in my apartment!

Dev: image selection from device

Creative: add a logo to the website (the thing that shows in the tab and other places)

See you soon! :)