Image projection!

tl;dr

Visit corner.builders to project a colorful star on the wall, that stays undistorted from some viewpoint (the one from the article picture)! Or look down the article for a video showing it on the live website in a regular mobile browser!

Where we are

As a reminder, the introduction post of this project

A recap of what we did so far, that can be read from the previous editions:

Made a quick demo video with a simple rectangle in a room.

Registered a domain.

Built our web-infra on aws

Build our CI/CD pipeline to deploy our code

Integrated some GUI

Defined the core user logic

and now...

Goals of this session

This session will be a little bit more dense than the other. My bad to not split earlier. What we want to achieve:

Implement our projection function and decide where to run it (CPU vs GPU?)

Use an image instead of a basic rectangle

Download a sample image as example

The desired end-result: be able to project a "star image" on a rectangular room, that would appear undistorted from the given viewpoint.

Projection formula

You can skip this section it if you don't want to see simple linear algebra, but the main idea remains intuitive so it's also easy to explain and understand with words. More often than not, if you can clearly state your problem, you are already halfway to the solution. Here the problem is simple to state: given that the photons travel in straight lines, the "depth" of the photon source is irrelevant, only it's direction matters. So in order for our image to appear as if it was printed on paper, we can just project from a point (an eye) through each pixel surface in a straight line and find the intersection with a wall, and draw our surface there. (of course this only works 1 eye, and our stereovision will tell us the depth of the surface, but it should work quite nicely as an approximation).

So if we call E the 3d eye position (so-called point of view), P the point to project (a pixel corner of the floating image), C the corner of a wall, U and V two perpendicular vectors describing a surface, then we have

E+t*(P-E)=C+aU+bV

The left-hand side E+t(P-E) is the point starting from the eye, going through point with the direction (P-E). The right hand side C+aU+bV is a way to describe a point on a plane. If those two points are the same, then it means we have a point on the projection line, belonging to the plane! The 3 things that are unknown is the triplet (a,b,t), rewriting the equation:

[U;V;E-P]x=E-C

Let's call A the 3x3 matrix [U;V;E-P], and x our triplet of interest, and B=E-C, then we have

Ax=B, that we can solve by inverting A : x=A.inv()*B. We can then retrieve the projected point with a side of our equation, for example E+t*(P-E) with t the third component of x.

And that's really it!

Now let's thing how/where to implement it.

CPU vs GPU

GPU computing excels in simple, highly parallel and independent tasks which fits our use case of projecting pixels.

Even though interacting with the GPU on Rust has been made quite easy with the help of some crates like glow or wgpu, it's still more verbose, boillerplatey than CPU programming.

We have also an advantage that we can exploit: while we are adjusting the viewpoint position/orientation we still dont need to project, because projected or not, the image will still appear the same (duh, that the all point). So we can only project as a 1time function when we validate the viewpoint, recompute, and resend the mesh to the GPU, and go into free-cam mode, and stay as a "floating rectangle image" while adjusting. This way the projection is a 1 time cost and should not impact user experience much (we really want it fluid while adjusting, and can accept some hypothetical "lag" after validating if it's not more than 1s). While it's tempting to go straight to GPU vertex shader implementation, I would call this unneeded premature optimization. So let's start with CPU, and switch to GPU if needed (for projection only, the rendering pipeline still goes through the gpu!).

The image grid mesh

Ok so we have a projection function. But what do we project exactly? We can represent our floating image as a triangular mesh, and our image as the texture. Each vertice has coordinates in the image texture, and keeps them (the texture coordinates) in the projected mesh. The fragment shader in the GPU will take care of color interpolation in the projected mesh.

Getting the image

For now (for simplicity) we won't have a choice in the image and get an example image from our server. Async operation like downloading a resource can be quite tricky in the single-threaded not async trunk WASM context without blocking the UI. A very nice crate to go around this limitation is poll_promise.

Recap

We can now instead of just viewing our floating rectangle, actually project an image of a star on the walls..!

See here for a demo video, or visit yourself corner.builders!

What's next?

Let's add a function to extract the projected image as images to print and try it in real-life!

See you soon! :)