Visit the About Me page for more extensive information on my current projects and experience
Posted: 2015-02-23 00:00:00

Implementing Screen space ambient occlusion in GameMaker Studio

Screen space ambient occlusion (SSAO), is an effect which aims to approximate global light obscurance in a 3D scene.

I started working on an SSAO shader in November 2014 with Andrew Hamilton (Orange451). It was an effect we had desired to have in our games, and whilst Andrew had come up with a technique in GM8 which involved using baked occlusion maps, it did not scale at all well if your scene had a substantial amount of geometry.

A more technical explanation of our shader is further down the page.


Download the demo: Download SSAO Demo
Get the shader: GMC Marketplace discussion topic

SSAO Attempt #1

The first really fantastic source we found on implementing SSAO was on a blog written by John Chapman ( His implementation of SSAO looked fantastic, and made use of a Normal-oriented hemisphere to avoid objects casting SSAO samples back into the object.

We had mixed results implementing the shader using this method in that the AO of the scene changed as the camera was rotated. Static scenes looked quite good, however there were too many visual artefacts for the shader to be at all useful.

This screenshot was a pic of the very first attempt we had after finally getting something to visually work. When the same effect was tested with an improved depth buffer, it didn’t look too bad. Though the rotation issue was still a very real problem.

At this time, we had also rendered our normal buffer incorrectly.

Whilst the AO doesn’t look bad here, this picture can help highlight some of the issues we were having. If you look at the edge of the teapots, and then at any of the walls, you can see the clear inconsistencies in what the Ambient Occlusion effect should look like.

We did spend a few more weeks continuing to work on this implementation, it was “almost” working, but after wasting countless hours, we decided the right choice was to abandon this version, and try again.

SSAO Attempt #2

The second attempt at Ambient occlusion wasn’t far off the first. We tried using the same principle, however changed some of the internal techniques. We were hoping to find that we had just made some mistake in one of the smaller steps that was resulting in the issues we were having.

Sadly, this was not the case, and we essentially ended up in the same situation again. It was clear at this time that the best option was to try and source alternative methods.

If there is one thing I have learnt as a graphics programmer, it is that there are only about 3 people on the internet who actually have a clue what they are doing: John Chapman, MJP and Louis Bavoil, the rest of us are just hacking away hoping that our shaders will work. This becomes clear when you start doing graphics research and find how many half complete solutions are posted online, or when people post sample code which works, but produces results nowhere near as good as the results they have in their pictures…

You would not believe how long it took to find accurate and reliable information on coordinate reconstruction from depth, there were so many places with straight-up mathematically wrong information, and then the wrong information duplicated and spurred out in other topics.

SSAO Attempt #3

At this point, we began looking into other SSAO implementations. This time however, we did not copy a singular implementation, but instead took ideas and concepts and used them to build our own method.

Much like any other method, the new method involved using a sampling kernel to project samples into geometry, compare their depth against a pre-rendered depth buffer to verify if a point occluded another point on the screen and so on.

It didn’t take too long to get a basic implementation working:

For this implementation we used a 2D sampling kernel as opposed to a 3D one and performed our coordinate transformations first in clip-space rather than in view-space.

The effect still had many problems at this point, however the one key difference was that we had a much better understanding of how to work with this shader, as it had been written from scratch. Andrew did most of the work on setting up the initial shader, however I then helped with fixing some of the artefacts the shader had.

Fixing the issues in the final implementation

This was probably one of the most challenging and time consuming aspects of the entire project. We had a working effect in-which the AO did not change as the camera was rotated, however it wasn’t game-ready.

The blur and noise crawling problem

One of the most prevalent problems was the initial implementation of a blur. To improve the visual result, SSAO implementations often make the effect look noisy. This is done to reduce visual banding in the effect which can look weird in games.

The first blur implementation was quite basic, and only performed a simple depth check to attempt to avoid making the whole diffuse texture look blurred in areas with AO. There also existed a problem of crawling noise such that when you moved the camera around, the SSAO’s noise would flicker. This was more visible at a distance.

To solve this, Andrew managed to implement a more advanced Bilateral blur which used per-fragment normal and depth information to filter out regions where blurring should and shouldn’t occur. This improved the visual quality of the image significanty.

The noise was also changed to a tiled method in which the noise gets tiled over the screen rather than determined at a per-pixel level. This solved the issue of crawling noise and ensured the noise was always consistent for any point on screen.

Quality vs performance issues

One of the next significant issues was to do with the overall performance of the shader. Unfortunately, the shader did not run particularly well, and required a very high resolution in order to look good, this of course caused a pretty significant down-grade in the performance as we had the Depth and normal resolution set to 2.0x which meant 4 fragment shader runs for every pixel on screen.

To fix this, we spent time experimenting with the shader setup, and found that introducing a “power” setting lead to a much smoother overall look than the strength setting being used previously. This, combined with the improved blur meant that the effect still looked nice at a standard 1.0x resolution, resulting in a much needed performance boost.

At this point, the shader was nearing release-standard. All we thought we had left to do was build a test level, and package the shader up into an extension.

Screen edge problem

One problem that we only became aware of under a normal use-case when we had built the test level was the fact that when the SSAO radius was set to a higher value, areas around the edge of the screen would have a weird sudden cut-off where the AO effect would stop working. This made for an annoying visual problem when you moved around in a dense scene.

To solve this, our first thought was to simply render more pixels and give the edge some “padding”. Whilst this was perfectly possible, it would cause a small decrease in performance, so instead we modified our algorithm to essentially just discard any sample that was cast outside of the scene, and this significantly improved the result, making the screen edge problem almost non-existent, or at least very unnoticeable.

This is a screenshot from our final AO demo, which I am very proud of!

Final optimisations and post-release improvements

Upon releasing our demo, we were pleased to hear that the demo ran on a decent range of systems, though there were quite a few users on mid-range PCs who could not maintain 60 fps whilst running the demo. We found a couple of areas which could be improved upon, with only a slight reduction in performance:

1) Max radius clamping

This optimisation involved putting a clamp on the maximum size of the radius (relative to the screen). This meant as you got very close to the objects, the radius would reach a maximum size relative to the screen. We found that due to the way texture caching is done, performing a texture lookup at a relative screen distance too far away from a pixel would cause a drastic and sudden drop in performance.

This customisable clamp value allows this issue to no longer be a problem, and it does not affect the visual quality of the AO at a distance from objects.

2) Blur sample optimisation

The blur shader is in some cases more intensive than the actual SSAO shader. We found that we could reduce the number of per-fragment samples from 4 to 2 and get a decent performance increase on weaker PCs. This meant the final result didn’t look quite as good, however the result still looked nice.

So there you have it! A long and waffled description of the process of getting ambient occlusion to work in GameMaker!