2D Outlines in Unity

There are many techniques for adding outlines to objects in games but they each come with their own trade-offs. This is particularly true when it comes to creating dynamic outlines for flat 2D objects where the quality of the available approaches is often less than desirable. Most of the dynamic outlining techniques seem to have been developed primarily with 3D assets in mind, or have other shortcomings that make them undesirable for certain use cases. 

Tom Mathews and I had been looking for a solution to this problem for a while. For our use case we wanted something that worked with 2D characters making use of skeletal animation and mesh deformation and we wanted the outlines to be chunky and smooth.

We tried a variety of existing solutions but ran into problems with each of them. Here is a quick summary of the main issues that led us to look for an alternative:

  • Many 3D techniques make use of mesh normals and therefore aren’t applicable to 2D.
  • Screen space effects are difficult to customise for specific objects and good quality thick outlines are difficult to achieve.
  • Approaches that create multiple duplicated versions of the sprite do not work effectively when you want thick outlines as the duplicates begin to separate.
  • Shaders like this colour the inside of the geometry, hiding the artwork, especially when using thick outlines.

Our solution is to allow the artist to draw the outlines into the original artwork at a reduced alpha value. In engine the character is then rendered out with a shader utilising dual alpha cutoffs to allow the outline to appear only around the outside of the character. This approach gives the artist complete control over the outlines and allows the outline to be any thickness while remaining smooth. In this post I’m going to cover most of the technical details of the shader. For more information on the art work flow you can check out Tom’s art workflow guide.

birdbriantumblr1

Artwork requirements:

This technique requires the art you want to outline to be done in a specific way. This unfortunately means that it can not be applied to existing artwork without first modifying them. The process isn’t too complicated and the control we get over the outline as well as the quality of the final outline made the trade-off worthwhile.

To create compatible art you simply draw whatever outline you want into the original textures but set at a lower alpha value than the main artwork. In game a custom shader will cause this outline to appear at full opacity and behind the main texture, preventing unwanted inner lines from drawing over the top of the character. Unfortunately you will be able to see the inner in your animation package however you can reduce the visual impact those lines have by choosing a very low alpha.

Advantages:

This shader has a few key advantages over other solutions. The major advantage is in the quality of outlines produced when a thick outline is required.

Unlike other solutions the outline does not encroach on the original artwork. This approach also gives the artist total control over outline thickness as well as allowing for variable outline thickness at different points on the character.

Outline colours can be easily customised per character by using separate materials unlike in screen space solutions.

The technique should also be very fast, with no extra geometry created, only 1 texture sample required in each pass and no use of additional cameras or render targets.

Disadvantages:

The main disadvantage to using this shader is that the art must be specifically prepped for use with the shader as previously described.

Internal lines are visible when not using the shader, and therefore in any external animation package. This can be distracting, however the visual impact of this can be reduced by using a very low alpha value on the outline in the original texture since it will be overridden in engine anyway.

Blending artifacts can appear on the edges of the artwork if the outline colour in the texture does not match the neighbouring pixels of the main artwork. This can be solved by ensuring these pixels match. More details can be found in this slide of Tom’s guide.

Shader Implementation:

The full source for the shader is available here.

***Please note!*** This shader makes use of the Offset tag which seems to be misbehaving in Unity 5.5. I currently have a bug report open and am waiting for a solution from the devs. I will update this post when I have more information.
For now this shader should work fine on 5.4. I might look at an alternate solution that actually changes the vertex positions if this fails to be resolved, but I’d rather avoid that additional complexity if possible.

This version of the shader uses Unity’s standard lighting model, but it should be relatively easy to convert any other alpha cutoff shader to do something similar. Below I will go over the key points that you would need to add to your own shader to support this technique. Note that for shaders that are not surface shaders secondary passes are added differently than I have described.

The shader works similarly to a traditional alpha cutoff shader, but with two alpha cutoff values instead of one. The first cutoff works as normal, controlling what pixels are visible. The second cutoff is used to control what pixels constitute the outline. As an example, if your normal artwork is 100% opaque and your outline is 30% opaque you might set your normal cutoff to 0.8 and your outline cutoff to 0.25.

In unity you can achieve a dual cutoff by creating a default surface shader, duplicating the CGPROGRAM so you have two passes and adding a clip function to each to do the alpha clipping with the appropriate cutoff.  Something like the following:

clip (albedo.a – _Cutoff);

Note that two different cutoff values are defined in the full shader, one for each pass.

_Cutoff (“Main Alpha Cutoff”, Range(0,1)) = 0.5
_OutlineCutoff (“Outline Alpha Cutoff”, Range(0,1)) = 0.25

Make sure to reference the correct cutoff in each pass.

At this point the outline will still draw over the top of the rest of the artwork. To solve this we can add an Offset tag to the main artwork pass to force its z depth in front of the outline.

Offset -3, [_LineOffset]

Exposing the offset value as a parameter allows it to be tweaked for individual characters to ensure the outline works correctly for overlapping pieces in the case of a character using skeletal animation, or to customise the behaviour when two outlined characters overlap. The -3 controls how the offset behaves when the image is viewed at grazing angles. This feature isn’t very well documented but -3 seemed to work best for me. Feel free to experiment with different values. If you are only ever viewing the character from straight on then 0 is probably a better value.

The final step is to colour the outline. This step is technically optional, you could simply put whatever outline colour you want to use directly into the texture, however colouring the outline can be useful if you want to change the colour at runtime. It also allows you to have the colour of the outline in the texture match the colour of the nearby pixels in the main artwork which helps reduce some potential blending artifacts that can otherwise occur as previously mentioned. Tom explores two approaches to creating your art with matching outline colours here.

To implement this the albedo colour is set to the value of a property (_OutlineColor) rather than the colour derived from the texture.

o.Albedo = _OutlineColor.rgb;

Simple! There really isn’t much to it, but it is pretty effective.

Some minor things to note:
I recommend only applying a shadow caster pass to the outline pass (by adding “addshadow” to the surface pragma). Adding a shadow pass to the main pass can cause shadows to appear on the outline which looks strange.
Both passes have a “Cull Off” tag. This is optional and only present to allow the shader to work with Spine.

Here is a comparison of a character created in spine to work with this shader. The standard Unity shader using alpha cutoff (Left) and our dual cutoff outline shader (right).

shadercomparison

Potential extensions:

Making use of an alpha gradient in the outline would allow control over outline width without having to modify the texture. This could also allow for runtime animation of the outline width by animating the outline alpha cutoff.
Potentially possible to extend this technique to support alpha’d edges, but doing this might also require the outlines to be drawn at a higher alpha value.
At the moment the outline pass uses the same lighting as the main pass. This doesn’t need to be true. The outline could just as easily be unlit or incorporate any other effect that might be desired.

That’s pretty much all there is to it. I recommend checking out Tom’s art workflow reference for some more detail on setting up artwork to work with this shader. Hopefully someone will find this useful!

Building a Messaging System for Unity

I recently had occasion to build some sort of messaging system for Unity and I thought I’d write a little bit about the solution I came to. First some background on what other people seem to do and why I didn’t want to do those things.

Unity’s SendMessage

First off is Unity’s built in solution to messaging (kind of), SendMessage(). Docs on how to use it are here: http://docs.unity3d.com/ScriptReference/GameObject.SendMessage.html
This isn’t really a system at all but sounds like it might be what you want when you first start looking for a solution to this problem so I thought I’d cover it.
SendMessage() doesn’t seem like a good idea for several reasons. Most obviously it uses reflection so it is slow, but more importantly it references the method to call using a string so it is brittle. The string name also means you don’t get intellisense when typing it, you can’t follow references using Resharper/Codelens and you won’t get compiler errors if you make a mistake. It also requires you to have a reference to the object you are calling it on so it causes coupling too. Sure you could build a notification system to work as an intermediary and then use SendMessage to call a method with a particular name to avoid coupling but the other problems still stand.

String Messages

Not too many people seem to use systems based around SendMessage due to performance. The more commonly suggested approach is to combine a string identifier for the message and a central manager that uses delegates to call methods on objects interested in particular messages. If for example you want a method LevelStarted() to be called when a message “level start” is sent you would do something like:

MessageManager.AddListener("level start", LevelStarted);

Then when a message is broadcast the manager invokes the corresponding delegate, calling all the methods that have been registered using that string. This system avoids the coupling and brittle nature of the previous example, but with a string for a key it is still easy to misspell the name and end up listening for a message that will never come or broadcasting a message that no one is listening for.

Typed Messages

While working on OniKira at BatCat Games we used a generics based system. This project was built for use with the open source 2D engine Duality rather than Unity but the two engines both make use of a similar component based approach to scene management and scripting. In this system components that want to listen for messages simply implement an interface. Each message type must inherit from a base GameMessage class and can then carry data about whatever event has happened. As an example, if you wanted to send a message when an enemy was damaged you might use something like this:

public class EntityDamagedMessage : GameMessage
{
	public GameObject Entity;
	public int DamageDealt;
}

To send a message from a component you call an extension method, SendMessage().

this.SendMessage(new EntityDamagedMessage
	{ Entity = GameObject, DamageDealt = Damage })

SendMessage gets all components in the scene that implement the message handling interface and calls HandleMessage on them. To handle the message you would write something like this:

public class TestClass : Component, IHandlesMessages
{
	public void HandleMessage(GameMessage msg)
	{
	var message = msg as EntityDiedMessage;

	if(message == null) return;

	//Do whatever
	}
}

This system ditches strings for custom types and therefore gives compiler warnings if you mistype the message name when handling or sending messages. The system also requires no registration step to receive messages as Duality itself keeps track of the types and interfaces implemented by all components so they can be easily found by calling Scene.Current.FindComponentsOfType<IHandlesMessage>(). This obviously adds the constraint that only components registered in the scene can receive messages directly. Other custom types within the system must have messages forwarded to them by a component if they wish to handle messages. Unfortunately automatic registration seems to be less practical in Unity where FindObjectsOfType() is slow and does not support interfaces. Besides this however, this system is largely compatible with Unity and accomplishes most of the things I want my messaging system to.

The one gripe I had after using this implementation for a long time was how handling multiple types of messages on a single component quickly became messy. You ended up with a very large HandleMessage method full of casts and subsequent null checks to see if you were interested in this particular message. Something like this:

public class TestClass : Component, IHandlesMessages
{
	public void HandleMessage(GameMessage msg)
	{

		if(message is EntityDiedMessage)
		{
			var message = msg as EntityDiedMessage;
			//Do whatever
			return;
		}

		if(message is TestMessageA)
		{
			var message = msg as TestMessageA;
			//Do something else
			return;
		}

		if(message is TestMessageB)
		{
			var message = msg as TestMessageB;
			//You get the idea
			return;
		}
	}
}

My System

So given all this, I went about designing my own messaging system with the following goals in mind:
1. It shouldn’t be easy to mistype a message and then have the whole thing silently not work.
2. The message handling method should only be called if you are actually interested in the type of message that has been sent.
3. Automatic registration probably isn’t feasible in unity but registering should still be as simple as possible.

My solution looks like this:

Messages are defined by classes that inherit from an abstract GameMessage class so messages look like they do in the Duality system.

public class EntityDamagedMessage : GameMessage
{
	public GameObject Entity;
	public int DamageDealt;
}

To send a message you make a call to a static MessageCenter class with an instance of a message:

MessagingCenter.BroadcastMessage(new EntityDamagedMessage
	{ Entity = GameObject, DamageDealt = Damage });

So far so similar. Finally, to receive a message you register with the messaging centre, implement a generic interface of the type of message that you want to receive and implement its handle message method. To handle multiple message you simply implement multiple interfaces. Below is an example of a class that handles three messages:

public class TestClass : IReceivesMessages<EntityDamagedMessage>,
	IReceivesMessages<TestMessageA>, IReceivesMessages<TestMessageB>
{
	private void OnEnable()
	{
		MessagingCenter.RegisterReceiver(this);
	}

	private void OnDisable()
	{
		MessagingCenter.RegisterReceiver(this);
	}

	public void HandleMessage<EntityDamagedMessage>
	(EntityDamagedMessage message)
	{
		var damage = message.Damage;
		//Do whatever
	}

	public void HandleMessage<TestMessageA>(TestMessageA message)
	{
		//Do something else
	}

	public void HandleMessage<TestMessageB>(TestMessageB message)
	{
		//You get the idea
	}
}

So goals achieved! Messages are typed so you get compiler errors if you make subtle mistakes. Methods are called with the actual message type as the parameter, not a generic GameMessage object so you don’t have to cast it or check if it is null. Also, registration is only one line and only takes one parameter! Short of not having to do it at all it doesn’t get much simpler than that.

So how does this work?
I’m going to start with registering as this was originally more complex. My first pass required you to register once for each interface that you implemented. That seemed kind of unnecessary so instead you now just pass whatever object implements the interfaces and the messaging system works out what messages you are interested in:

private static CollectionByType _receivers = new CollectionByType();

public static void RegisterReceiver(object receiver)
{
	if (receiver == null)
	{
		Debug.LogWarning(
			"Attempted to regsiter Message Receiver but was null");
		return;
	}
	foreach (var type in receiver.GetType().GetInterfaces())
	{
		if(type.IsGenericType
			&& type.GetGenericTypeDefinition() == 
			typeof(IReceivesMessages<>))
		_receivers.AddItem(type, receiver);
	}
}

RegisterReceiver loops over all the interfaces on a type looking for any that are of type IReceivesMessages. For any it finds it then adds a reference to the object to a collection (CollectionByType) using the interface type as a key. CollectionByType is basically a convenience wrapper for a Dictionary<Type, List<object>>. When you add an item it stores that item in a list corresponding with the type you passed. It also handles validation to make sure the object is actually of that type and casts the items to the requested type for you when you retrieve them.

When you call BroadcastMessage(), the type of the message is inferred from whatever message object you pass. The correct list of registered receivers is retrieved from the collection using that type and then HandleMessage() is called on each of the objects in that collection.

public static void BroadcastMessage<T>(T message) where T : GameMessage
{
	if (message == null)
	{
		Debug.LogWarning("Attempted to broadcast message but was null");
		return;
	}

	var receivers = _receivers.GetItems<IReceivesMessages<T>>();

	foreach (var receiver in receivers)
	{
		if(receiver != null)
			receiver.HandleMessage(message);
	}
}

DeregisterReceiver() works the same way as RegisterReceiver() but calls _receivers.RemoveItem() instead.

Caveats

The main downsides of using typed messages rather than string messages in my experience are that new message types have to be created by a programmer and they are a bit harder to deal with in editors. The first one I can live with. Adding a new message type only takes a few seconds and on this project I’m the only one touching the actual Unity project anyway.
Specifying a message in the editor will come into play if I ever want some kind of component that could send any type of message. This would be useful if you wanted to create a TriggerComponent that can send some user defined message when an object enters its collider. With a string message system you just have a string field on the component and you are done. I imagine I could create some kind of property drawer that uses a dropdown of all types that implement IReceivesMessages to pick a type and then let you edit it but I haven’t had a chance to test that out yet. I’ll post it here if I ever do.

In its current state the system will probably get a bit confused if handling a message causes a new message handler to register or deregister itself. Doing this would modify the underlying collection while it is still being iterated over, causing an InvalidOperationException. I’ll probably have to address this at some point, but for now it isn’t a problem because I’m not using the messaging system to do anything too complicated and I also wanted to get the general idea across here without convoluting it. Ideally at some point I’ll write a bunch of tests for this whole thing and fix that problem as part of that.

Another potential cause of errors is that not having automatic registration means that if you forget to deregister a receiver before it is destroyed or deactivated it could cause unexpected behaviour. If your receiver is a Unity component then Unity’s error handling seems to be pretty good and will give you a debug error about accessing a GameObject that no longer exists. If you message receiver is not a unity component however that might be harder to spot. I’d love suggestions on how to tackle that one if anyone has any because nothing has sprung to mind as of yet.

Worth mentioning that I also have no idea how this will fare in terms of performance. That is another thing I will deal with when the time comes, if it ever does. If I get around to writing tests perhaps I’ll also benchmark this against some other common implementations for reference, but with the amount I use it in this project I don’t think it will be a bottleneck any time soon.