Tessellation 101 – Melting Shader Part 3

Tessellation Header - Melt Shader Part 3

By Hugo Scott-Slade, Technical Director

In this series of posts I will go through the stages necessary to build a shader than can ‘melt’ any mesh into the ground. There are a lot of tutorials out there which cover the basics of writing shaders but these posts will aim to cover some good techniques for taking your game to the next level. I will assume you have a basic knowledge of shaders and we will build on each step until you have a cool shader to play with. This week will focus on Tessellation.

Part 1 & 2 covered

  • Object Space vs World Space
  • Moving Verts in the shader
  • Supporting Unity’s PBR rendering with the moving verts, including updating the shadows to match your new shape
  • Reorienting Normals based on new shape.
  • Creating the melt ‘shape’
  • Changing the objects material properties for the melted area

Parts 3 will cover next week

  • Tessellating the mesh to create new verts for a smooth melt shape
  • Optimising your tessellation
  • Combining tessellation, vertex and surface shaders
  • Circumventing current Unity limitations for the above.
  • General bug tips and things you will probably come across in your workflow and how to fix them!

All shaders and scripts have been written in Unity 5.4 but should work in older/newer versions too. Some shader keywords have changed in 5.4 such as _Object2World to unity_ObjectToWorld. If that doesn’t mean anything to you don’t worry – it will.

Additionally, the source code for all shaders is commented outlining the steps but this post will give a greater depth of information (it has pretty pictures) so be sure to read both. Also I won’t be using any of those fancy greek letters or reducing things to simple letter notation. I find that kinda stuff just bounces off my brain so I will aim for clarity over brevity.


Tessellation is really quite simple to get started with in Unity, they have a great library of functions in Tessellation.cginc and a lot of information in their manual. Tessellation is only supported on some of the latest hardware but it can really add to the quality of shader effects. If your machine supports it I would recommend looking into it as it can be great fun and will give you a lot of ideas.

Tessellating the melt shader

I will start by adding tessellation to a shader from the last part in this series. For reasons that will become clear later on I will start with the melting verts (3 - Melting Verts/Base) shader. Also for simplicity’s sake in the shaders I have written for this post I have not included a fallback SubShader if the machine running the shaders don’t support tessellation but that is easy to add for production.

Much like adding the custom vertex shader before the surface shader, we are going to add the tessellate shader before the vertex shader in the same manner.

#pragma surface surf Standard fullforwardshadows vertex:disp addshadow tessellate:tessDistance nolightmap

The tessellate parameter is identical to the vertex parameter in that it takes the name of a function that is setup with the correct arguments. Additionally I have added nolightmap as no second UV channel used and this is more efficient. In similar fashion you need to declare your own appdata struct in order to use tessellation so you are encouraged to only include the data you actually use for effeciency. If you don’t use a custom appdata struct you will get an error and the shader will not compile. I will include a list of common gotchas like this at the end of the post.

float4 tessDistance(appdata v0, appdata v1, appdata v2)
    float minDist = 10.0;
    float maxDist = 25.0;
    return UnityDistanceBasedTess(v0.vertex, v1.vertex, v2.vertex, minDist, maxDist, _Tess);

This tessellation function is taken directly from the Unity manual and controls the tessellation based on how far the vertex is from the camera. It’s a simple function which you can dissect if you look at the built in shader code (if you haven’t already, download this immediately from Unity). Adding the _Tess property and updating your vertex shader to use the new appdata struct then moving your scene camera in and out you can see how the vertices are created.


You could be tempted to slide that _Tess slider up to maximum and consider it a job done, everything is smooth and it looks great. I mean you totally could but you’re making a mammoth of new vertices all over your object and doing nothing with them. Any vert about the melt line is now split into 100s of new ones and below the surface you’re making many more you’re not even seeing. As with all things optimisation is a must. A simple optimisation is to check if any of our triangle’s verts are within the ‘melt’ range and if not don’t create any new verts.

I did this by taking Unity’s distance based tessellation method and adding my own conditions to it. The distance function is already calculating world space for each vert in each triangle it’s checking so before it returns a value I check if the melt value for that world position is outside of the melt range (+ a small threshold). Any vert outside the range is treated like it’s in the distance – no new verts are created. More could be done to improve the tessellation such as only creating verts along the silhouette of the shape but I will leave that to you.

Tessellation, Vertex and Surface

Unity does a great job for the most part at letting us write these small little surface shaders and then expanding them out to work across all platforms. Unfortunately it doesn’t work for all situations. It turns out that currently Unity won’t recognize your vertex shader if it has an out Input o if you use tessellation. Without manually adding the code in for each pass and each variation it is actually possible for us to create our melt shader. It’s not the most efficient method to do this but when developing shaders I care less about efficiency and more about iterations and workflow.

There are several variables Unity can pre-fill for you inside the Input struct without you having to manually fill them inside your vertex shader. The full list can be found here but for our purposes we only care about worldPos. The deformation is still handled within our vertex shader but in order to create the hard wave-like edge from the end of the part 2 we need a few things; the melt value and our object space position.

I created a new function called getMelt which takes a world space position and returns either a 0 or 1 for use within the surface shader.

float getMelt( float3 worldSpacePosition )
    float4 objectSpacePosition = mul( unity_WorldToObject, float4( worldSpacePosition, 0 ));
    float melt = ( worldSpacePosition.y - _MeltY ) / _MeltDistance;

    melt = 1 - saturate( melt );
//  melt = pow( melt, _MeltCurve ); // we don't care about the curve for this, just the linear melt value

    // this is the same code as the pixel shader in part 2
    float wave = sin( objectSpacePosition.x * 4 + objectSpacePosition.z * 5 ) * 0.15;
    float hardMelt = step( 0.5, melt + wave );

    return hardMelt;

Tessellation Final

And there it is, we can melt objects and create a nice smooth melt puddle using tessellation. There is more that will be done on this shader but I think this should give you a good process and enough building blocks to get to work on your own amazing shaders. A lot of the times I come across a cool shader online it is in its complete state with little to no explanation of process so I hope this helps you all find clarity.

Download the unity package for this post here

For more information about Cone Wars be sure to subscribe to the mailing list and follow @Glitchers on twitter.

Bonus – Common Issues

There are a few shader compilation bugs I got when writing this post so I wanted to quickly list out a few things I found and their solutions.

  • Shader error in 'Custom/5 - Tessellation/Base': invalid subscript 'texcoord4' at line 119 (on glcore) – you need to ensure you are using a custom appdata struct and defining that in your shader.
  • Shader error in 'Custom/6 - Tessellation + Pixels/Base': 'disp': no matching 1 parameter function at line 317 (on glcore) – your vertex shader function has the out Input o as a parameter which is currently not supported by Unity
  • Shader error in 'Custom/6 - Tessellation + Pixels/Base': Unexpected identifier "appdata". Expected: ')' at line 99... etc – you are using a custom appdata struct but it’s either not defined or not defined early enough