GLSL Programming/Blender/Cutaways

This tutorial covers discarding fragments, determining whether the front face or back face is rendered, and front-face culling. This tutorial assumes that you are familiar with varying variables as discussed in the tutorial on a RGB cube.

The main theme of this tutorial is to cut away triangles or fragments even though they are part of a mesh that is being rendered. The main two reasons are: we want to look through a triangle or fragment (as in the case of the roof in the drawing to the left, which is only partly cut away) or we know that a triangle isn't visible anyways; thus, we can save some performance by not processing it. OpenGL supports these situations in several ways; we will discuss two of them.

Very Cheap Cutaways

The following shader is a very cheap way of cutting away parts of a mesh: all fragments are cut away that have a positive $y$ coordinate in object coordinates (i.e. in the coordinate system in which it was modeled; see “Vertex Transformations” for details about coordinate systems). Here is the vertex shader:

         varying vec4 position_in_object_coordinates;
 
         void main()
         {
            position_in_object_coordinates= gl_Vertex;
            gl_Position = gl_ModelViewProjectionMatrix * gl_Vertex;
         }

And here is the fragment shader:

       
         varying vec4 position_in_object_coordinates;

         void main()
         {
            if (position_in_object_coordinates.y > 0.0) 
            {
               discard; // stop processing the fragment 
                  // if y coordinate is positive
            }
            if (gl_FrontFacing) // are we looking at a front face?
            {
               gl_FragColor = vec4(0.0, 1.0, 0.0, 1.0); // yes: green
            }
            else
            {
               gl_FragColor = vec4(1.0, 0.0, 0.0, 1.0); // no: red
            }
         }

When you apply this shader to any of the default objects, the shader will cut away half of them. This is a very cheap way of producing hemispheres or open cylinders. You will also notice that the “inside” of the object is invisible. This is due to back-face culling, which we discuss further below.

Discarding Fragments

Let's first focus on the discard instruction in the fragment shader. This instruction basically just discards the processed fragment. (This was called a fragment “kill” in earlier shading languages; I can understand that the fragments prefer the term “discard”.) Depending on the hardware, this can be a quite expensive technique in the sense that rendering might perform considerably worse as soon as there is one shader that includes a discard instruction (regardless of how many fragments are actually discarded, just the presence of the instruction may result in the deactivation of some important optimizations). Therefore, you should avoid this instruction whenever possible but in particular when you run into performance problems.

One more note: the condition for the fragment discard includes only an object coordinate. The consequence is that you can rotate and move the object in any way and the cutaway part will always rotate and move with the object. You might want to check what cutting in view or world space looks like: change the vertex and fragment shader such that the view coordinate $y$ or world coordinate $y$ is used in the condition for the fragment discard. Tip: see the tutorial on shading in view space for how to transform the vertex into world space.

Better Cutaways

You might try the following idea to improve the shader: change it such that fragments are discarded if the $y$ coordinate is greater than some threshold variable. Then introduce a game property to allow the user to control this threshold. Tip: see the tutorial on shading in view space for a discussion of game properties.

Back-Face Culling

By default, the Blender game engine only renders front faces because the inside of objects is usually invisible; thus, back-face culling can save quite some performance by avoiding to rasterize these triangles. This is achieved as follows: Triangles and vertices are processed as usual. However, after the viewport transformation of the vertices to screen coordinates (see “Vertex Transformations”) the graphics processor determines whether the vertices of a triangle appear in counter-clockwise order or in clockwise order on the screen. Based on this test, each triangle is considered a front-facing or a back-facing triangle. If it is back-facing and culling is activated it will be discarded, i.e., the processing of it stops and it is not rasterized.

However, we are able to see the inside with our shader because we have discarded some fragments; thus, we should deactivate back-face culling. In Blender 2.6, you should first set the render engine to Blender Game in the Info window. Then select an object in the 3D View and in a Properties window open the Material tab and make sure that the object has a material. Under Game Settings in the Material tab you can then deactivate Backface Culling. With the shader from above, the back faces should now appear in red.

Distinguishing between Front and Back Faces

As demonstrated by the shader, a special boolean variable gl_FrontFacing is available in the fragment shader that specifies whether we are looking at the front face of a triangle. Usually, the front faces are facing the outside of a mesh and the back faces the inside. (Just as the surface normal vector usually points to the outside.) However, the actual way front and back faces are distinguished is the order of the vertices in a triangle: if the camera sees the vertices of a triangle in counter-clockwise order, it sees the front face. If it sees the vertices in clockwise order, it sees the back face.

Our fragment shader checks the variable gl_FrontFacing and assigns green to the output fragment color if gl_FrontFacing is true (i.e. the fragment is part of a front-facing triangle; i.e. it is facing the outside), and red if gl_FrontFacing is false (i.e. the fragment is part of a back-facing triangle; i.e. it is facing the inside). In fact, gl_FrontFacing allows you not only to render the two faces of a surfaces with different colors but with completely different styles.

Note that basing the definition of front and back faces on the order of vertices in a triangle can cause problems when vertices are mirrored, i.e. scaled with a negative factor. Blender tries to take care of these problems; thus, just specifying a negative scaling in the Transform parameters of the object will usually not cause this problem. However, since Blender has no control over what we are doing in the vertex shader, we can still turn the inside out by multiplying one (or three) of the coordinates with -1, e.g. by assigning gl_Position this way in the vertex shader:

            gl_Position = gl_ModelViewProjectionMatrix 
            * vec4(-gl_Vertex.x, gl_Vertex.y, gl_Vertex.z, 1.0);

This just multiplies the $x$ coordinate by -1. For a sphere, you might think that nothing happens, but it actually turns front faces into back faces and vice versa; thus, now the inside is green and the outside is red. (By the way, this problem also affects the surface normal vector.) Thus, be careful with mirrors!

Summary

Congratulations, you have worked through another tutorial. (If you have tried one of the assignments: good job! I didn't yet.) We have looked at:

How to discard fragments.
How to deactivate the default culling of back faces.
How to render front-facing and back-facing triangles in different colors.