Erik Hausner
Technical Artist
Stylized Shader
Motivation:
For a few years I have tried and theorized on how to make stylized shaders that more closely resembles the one you find in stylized digital art such as this piece from Anton Fadeev. Because of this I wanted to try my hands at creating a stylized shader that would replicate one of the more painterly (but still digital) styles of art.
Vision:
As much as I would have wanted to fully recreate the style in this image, that was far outside the scope of this project. Luckily I was recommended to try the Kuwahara filter, which I had previously put of because of its square- like artifacts, but after digging I found the anisotrophic version with improved edge preservation. This reignited my hopes for the Kuwahara filter, and it showed potential to achieve a style I wanted.
Art piece from Anton Fadeev - Artstation
Anisotrophic Kuwahara Filter
How Kuwahara filter works:
The basic Kuwahara filter looks at 4 zones around the rendered pixel, finds the avrage color in each zone, compares that color to the renderd pixel and sets its color to the avrage of the zone with lowest diviation. This creates quite bad square like artifacts though because of the square shape of the zones, to fix this you can make the zones round and increase the zones to 8. This is pretty much the generalized Kuwahara filter.
Edge preservation:
As much as the original Kuwahara filter is meant to be edge preserving it looses quality when areas get to detailed, especialy when you upp the kernel size. To fix this you can first calculate the images Eigenvectors (pretty much the difference between the rendered pixel and its 4 neighboring pixels). This can then be used to find the angle of the pixel (based on difference in value). With this information you can then stretch the kernel in the direction of the pixel limiting bleed across edges and preserving edges.
Eigenvector generation
Kernel alignment to eigenvectors
Implementing the filter:
Implementing this filter requires quite a lot of heavy math, so luckily for me I found a video by Acerola where he documented the process as well as linked an implementation in Unity. To make this shader in Unreal I had to translate the Unity shader to the Unreal material graph and because of the repeated texture sampling as well as math I make the shader inside custom nodes, which meant writing the code in hlsl.
Using render targets:
Since the shader was divided into 2 main parts, the Eigenvector calculation and the Kuwahara filter, which both needed to sample neighboring pixels to get their value, I had to optimize by dividing the shader into 2 passes. The initial plan for this was to use multiple post processing materials, but since the Kuwahara filter required 2 textures, the scene texture and the eigenvector texture, I had to use a render target to get them both into the shader.
Better preview on blueprintue
Comparison
Dessert environment by Nighttimes on Fab
Stylized Shadows
Finding The edge:
To recreate the stylized look of many digital paintings I also wanted to replicate the look of saturated shadow edges. To do this I got the approximated lighting of the scene by subtracting the diffuse from the scene color. By using the scene lighting I was able to mask out the color from the shadow transition inside the scene.
Saturation Mask
Result
