Vertex Displacement – Melting Shader Part 1

Vertex Displacement Header - Melt Shader Part 1

By Hugo Scott-Slade, Technical Director

In this series of posts I will go through the stages necessary to build a shader than can ‘melt’ any mesh into the ground. There are a lot of tutorials out there which cover the basics of writing shaders but these posts will aim to cover some good techniques for taking your game to the next level. I will assume you have a basic knowledge of Unity, vertex shaders and surface shaders. We will build on each step until you have a cool shader to play with.

Part 1 will cover:

  • Object Space vs World Space
  • Moving Verts in the shader
  • Supporting Unity’s PBR rendering with the moving verts, including updating the shadows to match your new shape
  • Reorienting Normals based on new shape

Parts 2 & 3 will cover

  • Creating the melt ‘shape’
  • Changing the object’s material properties for the melted area
  • Tessellating the mesh to create new verts for a smooth melt shape
  • Optimising your tessellation
  • Combining tessellation, vertex and surface shaders
  • Circumventing current Unity limitations for the above
  • General bug tips and things you will probably come across in your workflow and how to fix them!

NB: All shaders and scripts have been written in Unity 5.4 but should work in older/newer versions too. Some shader keywords have changed in 5.4 such as _Object2World to unity_ObjectToWorld. If that doesn’t mean anything to you don’t worry – it will.

Additionally, the source code for all shaders is commented outlining the steps but this post will give a greater depth of information (it has pretty pictures) so be sure to read both. Also I won’t be using any of those fancy greek letters or reducing things to simple letter notation. I find that kinda stuff just bounces off my brain so I will aim for clarity over brevity.

Moving a vertex

Without any further delay let’s get to the fun stuff. This is the basic vertex shader created when you make a new Unlit shader (with the fog stripped out for clarity).

v2f vert (appdata v)
v2f o;
o.vertex = mul(UNITY_MATRIX_MVP, v.vertex);
o.uv = TRANSFORM_TEX(v.uv, _MainTex);
return o;

The line we care about is o.vertex = mul(UNITY_MATRIX_MVP, v.vertex);. This takes each vertex on the mesh and turns it into a point on the screen ready for the fragment to fill in. MVP stands for Model-View-Projection and is a matrix Unity provides us to do this common calculation. What we want to do is come in before the vert goes through this calculation. If we add a new line before the matrix multiplication we can start to see how we can play around with the verts.

v.vertex.x += 10;
o.vertex = mul(UNITY_MATRIX_MVP, v.vertex);

Simple Offset

Moving everything over to the side is great but we can do better. We can easily make our object ‘wave’, this is a good starter into vertex displacement. To get a nice wave shape the vertex shader should modify each vert in a different way, otherwise they’ll all move uniformly like the example above. To get our object to wave left and right as it goes up and down we will pass in the vertex’s y position. Using a sine wave gives a nice wave shape that repeats forever – very handy. Unity has a lot of built in variables for writing shaders. For animated effects you’re mostly going to use _Time (although don’t forget about _SinTime, _DeltaTime etc). _Time actually contains 4 different values. We will use _Time.y as this is the unmodified time in seconds so the most intuitive to work with.

float _Speed = 1;
float _Amount = 5;
float _Distance = 0.1;

v.vertex.x += sin( _Time.y * _Speed + v.vertex.y * _Amount ) * _Distance;


Object space vs World Space

That’s some basic deformation, but I’m sure a lot of you are wizards in waves already. Next thing I will cover quickly is World Space. We’ve been doing all our deformation in ‘object space’ or ‘model space’. If you run the pre above and scale the object the deformation will scale with it, if you rotate it the effect will rotate too.

Vertex Displacement Unlit

Here the sphere on the left is in object space and the sphere on the right is in world space. I am rotating each through script to show how each deformation is different. Going between object space and world space is a piece of cake.

float4 worldSpaceVertex = mul( unity_ObjectToWorld, objectSpaceVertex ); // object to world
float4 objectSpaceVertex = mul( unity_WorldToObject, worldSpaceVertex ); // world to object

Lighting deformation

Surface Displacement

Here are 3 weird wobbly metal spheres. Each is a step built on the previous and is integral to building a melt shader. They each use the same deformation pre as their unlit counterparts. To modify the verts of a lit shader we can create a custom vertex shader before the surface shader is run. Add vertex:methodName to the lighting pragma and the function in your code.

#pragma surface surf Standard fullforwardshadows vertex:vert

void vert( inout appdata_full v )
v.vertex.x += sin( _Time.y * _Speed + v.vertex.y * _Amount ) * _Distance;

Now you have the first sphere on the left. You will notice that the shadow of the left sphere doesn’t move like the other two. We need to let Unity know that we’ve moved the verts and to run their shadow pre through our function too. This is trivially easy so no excuses.

#pragma surface surf Standard fullforwardshadows vertex:vert addshadow

Spot the difference. That’s seriously it. Unity will read that and compile all the shaders you’ll ever need.

Ok, those 2 were easy. Too easy. If you take a look at the third and final sphere you can see the reflections of the world actually match up with the objects new shape (this is why I made them so grossly shiny). This is because we are recalculating the normals for each vertex in the vertex shader too.

Reorienting Vertex Normals

We’re gonna be breaking out some linear algebra for this next part. No need to dust off those those giant tomes from your formative years – this will be easy. We are going to create a new normal by creating 2 new verts in the vertex shader that are slightly offset across the surface of the object. First we will need to make our pre that moves the vert into a function so we can easily run it on other positions.

float4 getNewVertPosition( float4 p )
p.x += sin( _Time.y * _Speed + p.y * _Amount ) * _Distance;
return p;

To calculate the new positions we need to calculate the bitangent (sometimes called the binormal). The bitangent is a vector which is tangential to the normal of the vertex. That’s fancy talk for it’s sideways across the surface. It’s easiest to think of a normal at a vertex – a normal faces outwards from the surface of the object. A tangent and bintangent a both orthogonal to the normal and to each other. If you imagine a point on a plane with a normal that’s facing directly up with a value of ( 0, 1, 0 ) then the bitangent would be sideways across x with the value ( 1, 0, 0 ) and the tangent would be across z with the value ( 0, 0, 1 ). Here is a diagram to clarify.

Bitangent Diagram

We have access to the normal and the tangent in the vertex shader but no bitangent. Fortunately it’s just one method to calculate the bitangent.

bitangent = cross( normal, tangent );

The cross method with calculate the cross-product between the normal and the tangent which finds a vector that is orthogonal to both of the input vectors. If you calculate the cross product of up/down and forward/backwards, you would get left/right. There is a ‘handy’ rule to help remember the cross product. If you make a fist with your hand and stick your thumb up, your index finger forwards and middle finger sideways you have created 3 orthogonal vectors; the cross of any 2 of them will return the remaining.

Cross Product

Enough waving your hand about, we have all the pieces we need to create the new normal for our vertices. The trick is to create new a new tangent and bitangent. The tangent value passed to the shader indicates the surface direction before we deformed it. In order to find a new tangent and bitangent we will offset the input vertex position (note: not the displaced position) by a fraction of the tangent and bitangent.

float4 position = getNewVertPosition( vertex );
float4 positionAndTangent = getNewVertPosition( vertex + tangent * 0.01 );
float4 positionAndBitangent = getNewVertPosition( vertex + bitangent * 0.01 );

Then to create the new tangent and bitangent we just get the difference between the displaced verts.

float4 newTangent = ( positionAndTangent - position ); // leaves just 'tangent'
float4 newBitangent = ( positionAndBitangent - position ); // leaves just 'bitangent'

These values won’t be normalized but we aren’t setting the tangent of the vertex here, just the normal so we can actually save normalize instructions as both our new vectors have the same length, which is how much we offset the position by. To get the normal we just do the cross product of our new values and set it on the vertex.

float4 newNormal = cross( newTangent, newBitangent );
v.normal = newNormal;

That finishes of part 1, you know have the basis to do some great vertex displacement. You can move your verts and they will still be lit correctly. In the next post I will cover how I made the melt shader for Cone Wars. In part 3 we will improve it even further by adding tessellation and having super smooth curves on our shape as well as optimizing it and covering some of common pitfalls and how to fix them!

Download the unity package for this post here

For more information about Cone Wars be sure to subscribe to the mailing list and follow @Glitchers on twitter.