OpenGLES20 View is distorted with rotation matrix applied

Video of how it’s looking right now:

http://de.tinypic.com/r/2504bh5/8

I am applying a rotation matrix to my view matrix:

    float[] mTmpMatrix = new float[16];
    Matrix.setLookAtM(mTmpMatrix, 0, // mViewMatrix
            mCameraPosition[0], mCameraPosition[1], mCameraPosition[2], // eye
            mTargetRotPosition[0], mTargetRotPosition[1], mTargetRotPosition[2],
            0, 1, 0); // up
    Matrix.multiplyMM(mViewMatrix, 0, mRotationMatrix, 0, mTmpMatrix, 0);

To let the OpenGL view rotate respective to the device’s orientation to create some sort of augmented reality. The rotation matrix is calculated from sensor fusion (Magnetic, Accelerometer, Gyroscope) and is working quite well.

The problem I’m having right now is the following: The screen, better yet the objects I’m drawing, are distorted if I rotate my device, which should be a rotation along the Z axis.

The following images should give some sort of impression of what I’m talking about:

enter image description here

enter image description here

This is when looking “up”, at the ceiling of the rectangle I’ve drawn around the camera. The roof suddenly becomes a square, to still fit on the screen, but I don’t know why and how I can prevent this from happening, since if I, instead of rotating the view matrix, would rotate the lines, this should not happen.

For completion’s sake, the Line.class:

http://pastebin.com/GQszFEPm

Frustum settings (FoV for testing purposes, I did some experiments with this value, but no effects)

@Override
public void onSurfaceChanged(GL10 arg0, int width, int height) {
    this.width = width;
    this.height = height;
    GLES20.glViewport(0, 0, width, height);
    if (height > width)
        setFrustumVertical(100f, (float) height / (float) width, 0.01f, 11f);
    else setFrustumHorizontal(100f, (float) width / (float) height, 0.01f, 11f);
}

The helper methods, I’ve adapted from a tutorial:

private void setFrustumHorizontal(float fov, float aspect, float front, float back) {
    fov = (float) Math.toRadians(fov);                      // transform fov from degrees to radians
    fov = 2.0f * (float) Math.atan(Math.tan(fov * 0.5f) / aspect);  // transform from horizontal fov to vertical fov

    float tangent = (float) Math.tan(fov / 2.0f);               // tangent of half vertical fov
    float height = front * tangent;                 // half height of near plane
    float width = height * aspect;                  // half width of near plane

    setFrustum(-width, width, -height, height, front, back);
}

private void setFrustumVertical(float fov, float aspect, float front, float back) {
    fov = (float) Math.toRadians(fov);                      // transform fov from degrees to radians

    float tangent = (float) Math.tan(fov / 2.0f);               // tangent of half vertical fov
    float height = front * tangent;                 // half height of near plane
    float width = height * aspect;                  // half width of near plane

    setFrustum(-width, width, -height, height, front, back);
}

private void setFrustum(float left, float right, float bottom, float top, float near, float far) {
    Matrix.frustumM(mProjectionMatrix, 0, left, right, bottom, top, near, far);
}

Answer

I believe there is an issue with setFrustumVertical(). Let us assume your screen is 100 x 200 and in portrait (so twice as high as it is wide). Since height > width you will call:

setFrustumVertical(100f, 2, 0.01f, 11f)

If you dry run that method using those values (I’ve used degrees here rather than radians but the maths is the same):

fov = 100 degrees
tangent = tan(100 / 2) = 1.191
height = 0.01 x 1.191 = 0.01191
width = 0.01191 x 2 = 0.02382
setFrustrum(-0.02382, 0.02382, -0.01191, 0.01191, 0.01, 11)

Notice how the frustrum is twice as wide as it is tall? That’s back to front – it should be twice as high as it is wide in portrait. Switch width and height around:

float width = front * tangent;
float height = height * aspect;

Or perhaps width should instead be set with:

float width = height / aspect;

If you get a similar issue in landscape, dry run the other method by hand as well to ensure it creates a frustrum with the correct aspect ratio.