Camera2: Sensors + Layout Measurement + Animation of Capture button

In modern phone, we have many sensors in it. And we can use these sensors to create many interesting apps. Let's continue with my Camera2 project. We will use sensors to control the button.

git clone https://github.com/homandiy/Camera2TV Camera2Sensor

1, List of Sensors

First for all, we need to know the sensors in our phone. The SensorManager is in charge of all sensors.

// Sensor Manager
private SensorManager mSensorManager;

You need to initialize it in onCreate().

@Override
public void onCreate(Bundle savedInstanceState) {
    ...

    try {  
        mSensorManager = (SensorManager) getActivity()  
            .getSystemService(SENSOR_SERVICE);
    } catch (Exception e) {
        msg(getContext(), "Hardware compatibility issue");
    }

Now, you can reach the list.

private void getSensorList() {
    sensorList = mSensorManager.getSensorList(Sensor.TYPE_ALL);
    StringBuilder sensorText = new StringBuilder();

    for (Sensor currentSensor : sensorList ) {
        sensorText.append(currentSensor.getName()).append(
                System.getProperty("line.separator"));
    }

    ltag("sensors: "+sensorText);
}

Show them when the view is ready.

@Override
public void onViewCreated(@android.support.annotation.NonNull View view,
                          @Nullable Bundle savedInstanceState) {
    ...
    getSensorList();
}

Let's run. Turn on the logcat. You'd get the result.

I/MYLOG CameraFrag: sensors: LSM6DS3 Accelerometer
    LSM6DS3 Gyroscope
    YAS537 Magnetometer
    STK3013 Proximity
    YAS537 Uncalibrated Magnetometer
    LSM6DS3 Significant Motion Detector
    LSM6DS3 Uncalibrated Gyroscope
    LSM6DS3 Tilt Detector
    Screen Orientation Sensor
    Motion Sensor
    Samsung Game Rotation Vector Sensor
    Samsung Gravity Sensor
    Samsung Linear Acceleration Sensor
    Samsung Rotation Vector Sensor
    Samsung Orientation Sensor

My phone has 15 sensors in the list.

2, Rotation Vector Sensor

We need to know the orientation of a phone, so we can move the capture button. There are 4 vectors carried by sensor.

Take a look at the diagram on the left. We have x vector, y vector, z vector and m vector.

Let's design the UI of rotation vector,

rotation_sensor_data.xml

When you are done, you can include new XML files. Also, let's hide the TextureView to have clear view.

<com.huang.homan.camera2.View.common.AutoFitTextureView
    ...
    android:visibility="invisible"
    
<include
    android:id="@+id/tiltInclude"
    layout="@layout/acc_mag_sensor_data"
    android:layout_width="0dp"
    android:layout_height="wrap_content"
    android:visibility="visible"
    app:layout_constraintEnd_toEndOf="parent"
    app:layout_constraintStart_toStartOf="parent"
    app:layout_constraintTop_toTopOf="parent" />

Let's add some new variables into the CameraFragment.

// Rotation Sensor variables
@BindView(R.id.rvInclude)
View rvInclude;
@BindView(R.id.rvData1)
TextView rvData1;
@BindView(R.id.rvData2)
TextView rvData2;
@BindView(R.id.rvData3)
TextView rvData3;
@BindView(R.id.rvData4)
TextView rvData4;

Now, you need to initial at onCreate(); register the sensor at onStart(), onResume() and remove them at onPause().

@Override
public void onCreate(Bundle savedInstanceState) {
    ...

    try {
        ...
        mRotationSensor = mSensorManager
            .getDefaultSensor(Sensor.TYPE_ROTATION_VECTOR);

    } catch (Exception e) {
        ...
    }
}

@Override
public void onStart() {
    super.onStart();
    if (mRotationSensor != null) {
        mSensorManager.registerListener(
                this,
                mRotationSensor,
                SensorManager.SENSOR_DELAY_NORMAL);
    }
}

@Override
public void onResume() {
    super.onResume();
    mSensorManager.registerListener(
            this,
            mRotationSensor,
            SECOND);
}

@Override
public void onPause() {
    super.onPause();
    mSensorManager.unregisterListener(this);
}

Output the data to the layout:

@Override
public void onSensorChanged(SensorEvent sensorEvent) {


    if (sensorEvent.sensor == mRotationSensor) {
        if (sensorEvent.values.length > 4) {
            float[] truncatedRotationVector = new float[4];

            System.arraycopy(sensorEvent.values, 0, truncatedRotationVector, 0, 4);
            rvData1.setText(format("%.2f", truncatedRotationVector[0]));
            rvData2.setText(format("%.2f", truncatedRotationVector[1]));
            rvData3.setText(format("%.2f", truncatedRotationVector[2]));
            rvData4.setText(format("%.2f", truncatedRotationVector[3]));
        }
    }
}

Let's run. You can move the phone around to check how the numbers are changing.

3, Accelerometer and Magnetometer

You see. It's hard to set orientation by four vectors. So we need rotation matrix to make thing easier. The rotation matrix is required two sensors, Acceleromenter and Magnetometer.

To make a clear view, let's design another layout. The layout contains Azimuth which indicates direction in North/South/East/West, Pitch which indicates how phone tilts up/down, Roll which indicates how phone tilts left/right. Let's save it as acc_mag_sensor_data.xml.

Now, you can add tiltInclude to wrap with the new layout and move the rvInclude under the new layout in fragment_camera.xml.

<include
    android:id="@+id/tiltInclude"
    layout="@layout/acc_mag_sensor_data"
    android:layout_width="0dp"
    android:layout_height="wrap_content"
    android:visibility="visible"
    app:layout_constraintEnd_toEndOf="parent"
    app:layout_constraintStart_toStartOf="parent"
    app:layout_constraintTop_toTopOf="parent" />

<include
    android:id="@+id/rvInclude"
    layout="@layout/rotation_sensor_data"
    android:layout_width="0dp"
    android:layout_height="wrap_content"
    android:visibility="visible"
    app:layout_constraintEnd_toEndOf="parent"
    app:layout_constraintStart_toStartOf="parent"
    app:layout_constraintTop_toBottomOf="@id/tiltInclude" />

Next, let's add new variables in CarmeraFragment.

// Tilt sensors variables
@BindView(R.id.tiltInclude)
View tiltInclude;
@BindView(R.id.azimuthData)
TextView azimuthData;
@BindView(R.id.pitchData)
TextView pitchData;
@BindView(R.id.rollData)
TextView rollData;

As usual, you need to add them in onCreate(), onStart(), onResume(). I create a registerSensors() function for duplicated code.

@Override
public void onCreate(Bundle savedInstanceState) {
    ...
    try {
        ...
        mAccelerometer = mSensorManager
            .getDefaultSensor(Sensor.TYPE_ACCELEROMETER);
        mMagnetometer = mSensorManager
            .getDefaultSensor(Sensor.TYPE_MAGNETIC_FIELD);
    } catch (Exception e) {
        ...
    }
}

@Override
public void onStart() {
    super.onStart();        
    registerSensors();        
}

@Override
public void onResume() {
    super.onStart();        
    registerSensors();        
}

private void registerSensors() {
    ltag("Register Sensors");
    if (mRotationSensor != null) {
        mSensorManager.registerListener(
                this,
                mRotationSensor,
                SensorManager.SENSOR_DELAY_NORMAL);
    }
    if (mAccelerometer != null) {
        mSensorManager.registerListener(
                this,
                mAccelerometer,
                SensorManager.SENSOR_DELAY_NORMAL);
    }
    if (mMagnetometer != null) {
        mSensorManager.registerListener(
                this,
                mMagnetometer,
                SensorManager.SENSOR_DELAY_NORMAL);
    }
}

Now, let's output the data to the layout. We need two arrays to record the sensor data, and one for matrix.

/**
 * Sensors change
 */
private float[] mAccelerometerData = new float[3];
private float[] mMagnetometerData = new float[3];
private float[] orientationValues = new float[3];
@SuppressLint("DefaultLocale")
@Override
public void onSensorChanged(SensorEvent sensorEvent) { ... }

* Hard coded String: float format

Let's set all float to 2 decimals after period. You can open res/strings.xml to add a hard code string for the format, called float_format.

<string name="float_format">%1$.2f</string>

You can use the float_format in this way,

getResources().getString(R.string.float_format, %var% )

* Get data

We have 3 sensors in register. So we can use switch...case... in onSensorChanged().

int sensorType = sensorEvent.sensor.getType();
switch (sensorType) {
    case Sensor.TYPE_ACCELEROMETER:
        mAccelerometerData = sensorEvent.values.clone();
        break;

    case Sensor.TYPE_MAGNETIC_FIELD:
        mMagnetometerData = sensorEvent.values.clone();
        break;

    case Sensor.TYPE_ROTATION_VECTOR:
        float[] mRotationVectors = sensorEvent.values.clone();
        rvData1.setText(getResources().getString(
                R.string.float_format, mRotationVectors[0]));
        rvData2.setText(getResources().getString(
                R.string.float_format, mRotationVectors[1]));
        rvData3.setText(getResources().getString(
                R.string.float_format, mRotationVectors[2]));
        rvData4.setText(getResources().getString(
                R.string.float_format, mRotationVectors[3]));
        break;

    default:
        return;
}

Following the switch, you can get the rotation matrix.

// get matrix
getMatrix(mAccelerometerData, mMagnetometerData);
if (orientationValues != null) {
    // update tilt data
    azimuthData.setText(getResources().getString(
            R.string.float_format, orientationValues[0]));
    pitchData.setText(getResources().getString(
            R.string.float_format, orientationValues[1]));
    rollData.setText(getResources().getString(
            R.string.float_format, orientationValues[2]));
} else {
    // update tilt data with error
    azimuthData.setText(getResources().getString(
            R.string.error_matrix));
    pitchData.setText(getResources().getString(
            R.string.error_matrix));
    rollData.setText(getResources().getString(
            R.string.error_matrix));
}

Here is the getMatrix().

private void getMatrix(float[] mAccelerometerData, float[] mMagnetometerData) {
    float[] rotationMatrix = new float[9];
    boolean rotationOK = SensorManager.getRotationMatrix(rotationMatrix,
            null, mAccelerometerData, mMagnetometerData);

    if (rotationOK) {
        SensorManager.getOrientation(rotationMatrix, orientationValues);
    } else {
        orientationValues = null;
    }
}

Here is the error_matrix.

<string name="error_matrix">error</string>

4, Lock on Portrait

To prevent the screen automatically rotation, we need to lock it into portrait in the fragment.

@Override
public void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);

    getActivity().setRequestedOrientation(ActivityInfo.SCREEN_ORIENTATION_PORTRAIT);

5, Test Result

After you lock the screen, you can run the code.

Here is some read you will get:

North, Azimuth = 0
South, Azimuth = 3 or -3
East, Azimuth = 2
West, Azimuth = -2


Flat ground, Pitch = 0. 
Landscape Vertical: Pitch = 0.
Portrait Vertical: Pitch = -1.5. 
Portrait Vertical upside down: Pitch = 1.5

Face up: Roll = 0
Face down: Roll = 3
Landscape left upside: Roll = 1.3
Landscape right upside: Roll = -1.3

6, Setup Capture Button

According to the data, you need to set the capture button position.

Portrait: buttom right, 0 degree
Portrait upside down: top left and rotate 180
Landscape left upside: top right and rotate 90
Landscape right upside: buttom left and rotate 270 

* Remove Appbar

Appbar is ugly in camera app. Let's remove it in the activity and its layout.

* Get Layout Measurement

You need layout measurement to be a preset for the animation. Let's give the ID of most exterior layout of fragment_camera.xml.

<?xml version="1.0" encoding="utf-8"?>
<android.support.constraint.ConstraintLayout ...  

    android:id="@+id/cameraCL"

Here is the observer, called ViewTreeObserver in the CameraFragment.

@BindView(R.id.cameraCL)
ConstraintLayout cameraCL;
private int height;
private int width;
private int offset;

@BindView(R.id.captureIV)
ImageView captureIV;
private int capButHeight;
private int capButWidth;
private position capPosition = position.BottomRight;

@Override
public View onCreateView(...) {
    ...

    // Get fragment measurement
    ViewTreeObserver vto = cameraCL.getViewTreeObserver();
    vto.addOnGlobalLayoutListener (
            new ViewTreeObserver.OnGlobalLayoutListener() {
        @Overridepublic void onGlobalLayout() {
            cameraCL.getViewTreeObserver()
                    .removeOnGlobalLayoutListener(this);
            height  = cameraCL.getMeasuredWidth();
            width = cameraCL.getMeasuredHeight();

            ltag("fragment ~ h: "+height+"    w: "+width);

            capButHeight = captureIV.getHeight();
            capButWidth = captureIV.getWidth();

            ltag("capture button ~ h: "+capButHeight+"    w: "+capButWidth);
            int location[] = new int[2];
            captureIV.getLocationOnScreen(location);

            int offset1 = height - location[0];
            int offset2 = width - location[1];
            offset = offset1;
            ltag("capture button ~ x: "+location[0]+", offsetX: "+offset1+"."+
                              "    y: "+location[1]+", offsetY: "+offset2+".");

            createButtonMap(height, width, capButHeight, capButWidth);

            registerSensors();
        }
    });

    return view;
}

public enum position {
    TopLeft, TopRight, BottomLeft, BottomRight
}

private class ButtonLocation{
    private int x;
    private int y;

    public ButtonLocation(int x, int y) {
        this.x = x;
        this.y = y;
    }

    public int getX() {
        return x;
    }

    public int getY() {
        return y;
    }

    public String toString() {
        return "x: "+x+",  y: "+y;
    }
}

private Map<position, ButtonLocation> ButtonMap = new HashMap<>();

private void createButtonMap(int x, int y, int cbx, int cby) {
    int gap = offset - cbx;
    // set top left position
    ButtonMap.put(position.TopLeft, new ButtonLocation(gap, gap));
    // set top right position
    ButtonMap.put(position.TopRight, new ButtonLocation((x-offset), gap));
    // set bottom left position
    ButtonMap.put(position.BottomLeft, new ButtonLocation(gap, (y-offset)));
    // set top right position
    ButtonMap.put(position.BottomRight, new ButtonLocation((x-offset), (y-offset)));
}

You should register sensor in the observer instead of onStart(). Also, you need to set a test condition in onResume().

@Override
public void onStart() {
    super.onStart();
    ltag("onStart()");
}

@Override
public void onResume() {
    super.onResume();
    ltag("onResume()");
    if (height > 0) {
        registerSensors();
        // Reset button position
        captureIV.setX((float) ButtonMap.get(position.BottomRight).getX());
        captureIV.setY((float) ButtonMap.get(position.BottomRight).getY());
    }
}

Run. Here is the logcat.

I/MYLOG CameraFrag: Permission: true
I/MYLOG CameraFrag: Permissions granted!
I/MYLOG CameraFrag: onCreateView()
I/MYLOG CameraFrag: onStart()
I/MYLOG CameraFrag: onResume()
I/MYLOG CameraFrag: fragment ~ h: 720    w: 1280
I/MYLOG CameraFrag: capture button ~ h: 160    w: 160
I/MYLOG CameraFrag: capture button ~ x: 544, offsetX: 176.    y: 1104, offsetY: 176.
I/MYLOG CameraFrag: Register Sensors

This is the correct order in this project.

* Move Button without Rotation

We need data from pitch, roll, vector 1, and vector 3.

@SuppressLint("DefaultLocale")
@Override
public void onSensorChanged(SensorEvent sensorEvent) {

    ...

    // get matrixfloat[] mOrientationValues = getMatrix(mAccelerometerData, mMagnetometerData);if (mOrientationValues != null) {
        ...

        moveCaptureButton(pitch, roll, mVectors[0], mVectors[2]);

Insert the move command in the matrix.

private void moveCaptureButton(float pitch, float roll, float v1, float v3) {

    position newPos = getPosition(pitch, roll, v1, v3);

    if (newPos != capPosition) {
        ltag("Old Pos: "+capPosition.name()+"    --   New Pos: "+newPos.name());


        ButtonLocation oldBL = ButtonMap.get(capPosition);
        ButtonLocation newBL = ButtonMap.get(newPos);
        ltag("Old Location: "+oldBL.toString()+".  New Location: "+newBL.toString());

        capPosition = newPos;
        captureIV.setX((float) ButtonMap.get(newPos).getX());
        captureIV.setY((float) ButtonMap.get(newPos).getY());

        int location[] = new int[2];
        captureIV.getLocationOnScreen(location);
        ltag("Capture Button:  x: "+location[0]+"  y: "+location[1]);
    }
}

Now, you can run and test the position.

*Rotate the Button by position

captureIV.setRotation(0);
switch (newPos) {
    case TopLeft: captureIV.setRotation(180); break;
    case TopRight:  captureIV.setRotation(-90); break;
    case BottomLeft:  captureIV.setRotation(90); break;
}

Here is the update of getPosition(). Before you copy the code, you need to play the phone to gather all the data. Then you will understand why I use these conditions to check the orientation.

private position getPosition(float pitch, float roll, float v1, float v3) {
    boolean landscape = false;
    if (pitch > -0.5f && pitch < 0.5f) {
        if (roll < 0)
            return position.BottomLeft;
        else
            return position.TopRight;
    }

    if (v3 > 0)
         return position.BottomRight;
    else
        return position.TopLeft;
}

Finally, let's run again. The app should rotate button by following the orientation.

Github: https://github.com/homandiy/Camera2Sensors