ShaderLab: Stencil

The stencil bufferA memory store that holds an 8-bit per-pixel value. In Unity, you can use a stencil buffer to flag pixels, and then only render to pixels that pass the stencil operation. More info
See in Glossary can be used as a general purpose per pixel mask for saving or discarding pixelsThe smallest unit in a computer image. Pixel size depends on your screen resolution. Pixel lighting is calculated at every screen pixel. More info
See in Glossary.

The stencil buffer is usually an 8 bit integer per pixel. The value can be written to, increment or decremented. Subsequent draw calls can test against the value, to decide if a pixel should be discarded before running the pixel shaderA small script that contains the mathematical calculations and algorithms for calculating the Color of each pixel rendered, based on the lighting input and the Material configuration. More info
See in Glossary.

Syntax

Ref

    Ref referenceValue

The value to be compared against (if Comp is anything else than always) and/or the value to be written to the buffer (if either Pass, Fail or ZFail is set to replace). 0–255 integer.

ReadMask

    ReadMask readMask

An 8 bit mask as an 0–255 integer, used when comparing the reference value with the contents of the buffer (referenceValue & readMask) comparisonFunction (stencilBufferValue & readMask). Default: 255.

WriteMask

    WriteMask writeMask

An 8 bit mask as an 0–255 integer, used when writing to the buffer. Note that, like other write masks, it specifies which bits of stencil buffer will be affected by write (i.e. WriteMask 0 means that no bits are affected and not that 0 will be written). Default: 255.

Comp

    Comp comparisonFunction

The function used to compare the reference value to the current contents of the buffer. Default: always.

Pass

    Pass stencilOperation

What to do with the contents of the buffer if the stencil test (and the depth test) passes. Default: keep.

Fail

    Fail stencilOperation

What to do with the contents of the buffer if the stencil test fails. Default: keep.

ZFail

    ZFail stencilOperation

What to do with the contents of the buffer if the stencil test passes, but the depth test fails. Default: keep.

Comp, Pass, Fail and ZFail will be applied to the front-facing geometry, unless Cull Front is specified, in which case it’s back-facing geometry. You can also explicitly specify the two-sided stencil state by defining CompFront, PassFront, FailFront, ZFailFront (for front-facing geometry), and CompBack, PassBack, FailBack, ZFailBack (for back-facing geometry).

Comparison Function

Comparison function is one of the following:

Greater	Only render pixels whose reference value is greater than the value in the buffer.
GEqual	Only render pixels whose reference value is greater than or equal to the value in the buffer.
Less	Only render pixels whose reference value is less than the value in the buffer.
LEqual	Only render pixels whose reference value is less than or equal to the value in the buffer.
Equal	Only render pixels whose reference value equals the value in the buffer.
NotEqual	Only render pixels whose reference value differs from the value in the buffer.
Always	Make the stencil test always pass.
Never	Make the stencil test always fail.

Stencil Operation

Stencil operation is one of the following:

Keep	Keep the current contents of the buffer.
Zero	Write 0 into the buffer.
Replace	Write the reference value into the buffer.
IncrSat	Increment the current value in the buffer. If the value is 255 already, it stays at 255.
DecrSat	Decrement the current value in the buffer. If the value is 0 already, it stays at 0.
Invert	Negate all the bits.
IncrWrap	Increment the current value in the buffer. If the value is 255 already, it becomes 0.
DecrWrap	Decrement the current value in the buffer. If the value is 0 already, it becomes 255.

Deferred rendering path

Stencil functionality for objects rendered in the deferred rendering pathThe technique Unity uses to render graphics. Choosing a different path affects the performance of your game, and how lighting and shading are calculated. Some paths are more suited to different platforms and hardware than others. More info
See in Glossary is somewhat limited, as during the base pass and lighting pass the stencil buffer is used for other purposes. During those two stages stencil state defined in the shader will be ignored and only taken into account during the final pass. Because of that it’s not possible to mask out these objects based on a stencil test, but they can still modify the buffer contents, to be used by objects rendered later in the frame. Objects rendered in the forward renderingA rendering path that renders each object in one or more passes, depending on lights that affect the object. Lights themselves are also treated differently by Forward Rendering, depending on their settings and intensity. More info
See in Glossary path following the deferred path (e.g. transparent objects or objects without a surface shader) will set their stencil state normally again.

The deferred renderingThe process of drawing graphics to the screen (or to a render texture). By default, the main camera in Unity renders its view to the screen. More info
See in Glossary path uses the three highest bits of the stencil buffer, plus up to four more highest bits - depending on how many light mask layers are used in the sceneA Scene contains the environments and menus of your game. Think of each unique Scene file as a unique level. In each Scene, you place your environments, obstacles, and decorations, essentially designing and building your game in pieces. More info
See in Glossary. It is possible to operate within the range of the “clean” bits using the stencil read and write masks, or you can force the camera to clean the stencil buffer after the lighting pass using CameraA component which creates an image of a particular viewpoint in your scene. The output is either drawn to the screen or captured as a texture. More info
See in Glossary.clearStencilAfterLightingPass.

Example

The first example shader will write the value ‘2’ wherever the depth test passes. The stencil test is set to always pass.

Shader "Red" {
    SubShader {
        Tags { "RenderType"="Opaque" "Queue"="Geometry"}
        Pass {
            Stencil {
                Ref 2
                Comp always
                Pass replace
            }
        
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            struct appdata {
                float4 vertex : POSITION;
            };
            struct v2f {
                float4 pos : SV_POSITION;
            };
            v2f vert(appdata v) {
                v2f o;
                o.pos = UnityObjectToClipPos(v.vertex);
                return o;
            }
            half4 frag(v2f i) : SV_Target {
                return half4(1,0,0,1);
            }
            ENDCG
        }
    } 
}

The second shader will pass only for the pixels which the first (red) shader passed, because it is checking for equality with the value ‘2’. It will also decrement the value in the buffer wherever it fails the Z test.

Shader "Green" {
    SubShader {
        Tags { "RenderType"="Opaque" "Queue"="Geometry+1"}
        Pass {
            Stencil {
                Ref 2
                Comp equal
                Pass keep 
                ZFail decrWrap
            }
        
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            struct appdata {
                float4 vertex : POSITION;
            };
            struct v2f {
                float4 pos : SV_POSITION;
            };
            v2f vert(appdata v) {
                v2f o;
                o.pos = UnityObjectToClipPos(v.vertex);
                return o;
            }
            half4 frag(v2f i) : SV_Target {
                return half4(0,1,0,1);
            }
            ENDCG
        }
    } 
}

The third shader will only pass wherever the stencil value is ‘1’, so only pixels at the intersection of both red and green spheres - that is, where the stencil is set to ‘2’ by the red shader and decremented to ‘1’ by the green shader.

Shader "Blue" {
    SubShader {
        Tags { "RenderType"="Opaque" "Queue"="Geometry+2"}
        Pass {
            Stencil {
                Ref 1
                Comp equal
            }
        
            CGPROGRAM
            #include "UnityCG.cginc"
            #pragma vertex vert
            #pragma fragment frag
            struct appdata {
                float4 vertex : POSITION;
            };
            struct v2f {
                float4 pos : SV_POSITION;
            };
            v2f vert(appdata v) {
                v2f o;
                o.pos = UnityObjectToClipPos(v.vertex);
                return o;
            }
            half4 frag(v2f i) : SV_Target {
                return half4(0,0,1,1);
            }
            ENDCG
        }
    }
}

The result:

Another example of a more directed effect. The sphere is first rendered with this shader to mark-up the proper regions in the stencil buffer:

Shader "HolePrepare" {
    SubShader {
        Tags { "RenderType"="Opaque" "Queue"="Geometry+1"}
        ColorMask 0
        ZWrite off
        Stencil {
            Ref 1
            Comp always
            Pass replace
        }

        CGINCLUDE
            struct appdata {
                float4 vertex : POSITION;
            };
            struct v2f {
                float4 pos : SV_POSITION;
            };
            v2f vert(appdata v) {
                v2f o;
                o.pos = UnityObjectToClipPos(v.vertex);
                return o;
            }
            half4 frag(v2f i) : SV_Target {
                return half4(1,1,0,1);
            }
        ENDCG

        Pass {
            Cull Front
            ZTest Less
        
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            ENDCG
        }
        Pass {
            Cull Back
            ZTest Greater
        
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            ENDCG
        }
    } 
}

And then rendered once more as a fairly standard surface shaderUnity’s code generation approach that makes it much easier to write lit shaders than using low level vertex/pixel shader programs. More info
See in Glossary, with the exception of front face culling, disabled depth test and stencil test discarding previously marked pixels:

Shader "Hole" {
    Properties {
        _Color ("Main Color", Color) = (1,1,1,0)
    }
    SubShader {
        Tags { "RenderType"="Opaque" "Queue"="Geometry+2"}

        ColorMask RGB
        Cull Front
        ZTest Always
        Stencil {
            Ref 1
            Comp notequal 
        }

        CGPROGRAM
        #pragma surface surf Lambert
        float4 _Color;
        struct Input {
            float4 color : COLOR;
        };
        void surf (Input IN, inout SurfaceOutput o) {
            o.Albedo = _Color.rgb;
            o.Normal = half3(0,0,-1);
            o.Alpha = 1;
        }
        ENDCG
    } 
}

The result: