How do you estimate the essential and fundamental matrices from image pairs?

1 What is multi-view geometry?

Multi-view geometry is the study of how multiple images of the same scene are related by the geometry of the cameras and the scene. It deals with questions like: How can we recover the camera poses and the scene points from multiple images? How can we compare the views of the same point or line in different images? How can we determine if a point is visible or occluded in a given view? Multi-view geometry provides the tools and methods to answer these questions using concepts like homographies, epipolar geometry, and triangulation.

3 What are the essential and fundamental matrices?

The essential and fundamental matrices are two 3x3 matrices that encode the epipolar geometry between two images. The essential matrix E relates the normalized image coordinates of corresponding points in two cameras with known intrinsic parameters, while the fundamental matrix F links the pixel coordinates of corresponding points in two uncalibrated cameras with unknown intrinsic parameters. Both matrices have rank 2 and can be expressed through the equations x'TEx = 0 and x'TFx = 0 for any pair of corresponding points x and x' in the two images.

4 How do they relate to the camera poses and the scene geometry?

The essential and fundamental matrices contain information about the relative pose of the two cameras and the geometry of the scene. The essential matrix E can be decomposed into a rotation matrix R and a translation vector t that represent the relative orientation and position of the two cameras. The fundamental matrix F can be obtained from the essential matrix E and the intrinsic matrices K and K' of the two cameras as follows:

F = K' T EK

The scene geometry can also be inferred from the essential and fundamental matrices. For example, the epipolar lines can be computed from F as follows:

l = Fx

l' = F T x'

where l and l' are the epipolar lines in the first and second image, respectively. The epipoles can be computed from E or F as the null vectors of these matrices.

5 How can we estimate them from image pairs?

Various methods exist to estimate the essential and fundamental matrices from image pairs, depending on the available data and assumptions. The 8-point algorithm assumes at least 8 point correspondences and solves a linear system to find E or F, and then enforces the rank 2 constraint. The normalized 8-point algorithm improves accuracy and stability by applying a normalization transformation to the image coordinates before solving the linear system. The 7-point algorithm assumes exactly 7 point correspondences and solves a system of cubic equations to find F. The 5-point algorithm assumes at least 5 point correspondences and solves a system of polynomial equations to find E. Finally, the RANSAC algorithm is a robust estimator that can handle outliers and noise in the point correspondences. It randomly samples a subset of points, applies one of the previous methods, and evaluates the quality of the solution by counting inliers that satisfy the epipolar constraint.