Finished!

Well, I handed in my project on Tuesday, dissertation, source code and all. I can't say that I'm totally satisfied with my application, but I am quite pleased with the dissertation. All that's left is the showcase tomorrow and everything is totally finished.

ttfn

Progress Report ~7

Just thought I'd update with my dissertation progress. I'm a little bit behind schedule, I had hoped to finish the first draft before Easter, but as it stands...that didn't happen. I am over halfway through writing it though, and progress is relatively good. I have finished the introduction, the literature review and a few sections of my methodology section, such as the detailing of the blurs used and the input required for the ssao shader.

I am currently working on the Future Work section, and hope to get on to the rest of the Methodology section tomorrow. With any luck I will be finished the first draft by next Monday.

Dissertation: Lit Review: Screen Space Radiosity

2.4 Screen Space Radiosity

Chapter 6.4 of ShaderX7, Fast Fake Global Illumination, describes several techniques used to simulate the effects of indirect lighting. Included in these techniques is the author’s implementation of Screen Space Ambient Occlusion and its logical expansion into a fake radiosity effect, i.e. Screen Space Radiosity (SSRad). SSRad simulates a local radiance transfer (a fake radiosity), by using the colour information for the scene in the SSAO calculations. This requires the scene’s colours to be rendered to a texture, in much the same way as the depth and normals. The SSAO term will require three channels instead of one, i.e. RGB, but the actual occlusion calculation remains the same. The occlusion factor is combined with the occluder colour values, resulting in a local radiosity effect. [Briney et al, 2010]      

A similar approach was taken by a member of the GameDev forums, who posted a technique that they dubbed SSGI (Screen Space Global Illumination). This technique also uses a colour buffer, and multiplies the RGB values of the occluder with the AO term. [ArKano22, 2009] However, this implementation also stores the single channel AO term and when outputting the final pixel value, uses the AO term and the colour bleeding term added together. This allows the grey shadows of ambient occlusion and the local radiosity effect to be combined. 

Dissertation: Introduction

1. Introduction

Realism is a big issue in the games industry today, graphical realism in particular. Video games have evolved greatly over the past thirty or so years, from pixelated sprites and electronic blips to fully 3D interactive worlds. And as games get more and more realistic, the graphical expectations of the consumers are getting higher and higher. At the same time however, these gamers also want better performance – high frame-rates and fast loading times. So there is a lot of pressure on the industry to develop graphical techniques that look good without sacrificing performance.

When it comes to adding realism to 3D games, graphical elements such as lighting and shadows are at the forefront. Without the definition that lighting, whether it be point, spot, or directional, provides, 3D objects look flat and unrealistic. Shadows are also important in that they provide visual clues to the spatial relationships between object in the 3D scene. [Drettakis and Stamminger, 2002] The simulation of these effects – the effects of direct lighting – is standard practice in today’s computer games, using the Lambertian and Blinn models. [Filion, 2010] With that in mind, the next step along the path to realism is the simulation of indirect lighting, i.e. global illumination and ambient occlusion.  

“In reality, a significant amount of light in a scene comes from light reflecting from surfaces.” [Akenine-Moller and Haines, 2002, p277] Without simulating this reflected light, parts of the scene not directly affected by light sources would be left in complete darkness, which is not the case in real life. [Boonekamp, 2010] Global illumination (GI) calculations differ from direct illumination in that they use data gathered from surrounding objects to determine the lighting at a particular point, as opposed to needing only the surface data of that point. GI techniques simulate the ambient effects caused by light bouncing off of diffuse surfaces and around a scene, adding extra depth and definition to said scene. Unfortunately, such techniques i.e. ray tracing and radiosity, are computationally expensive and therefore not viable for use in real-time applications such as computer games. This means that other ways of producing similar effects must be found, that do not require so much processing power. Ambient occlusion is one such technique.

Ambient Occlusion (AO) is a subset of global illumination and the effect that it produces is produced automatically by techniques such as ray tracing. It is only an approximation of actual GI effects but can be implemented in real time in a far more cost effective manner than other, more accurate techniques. It is a simulation of the soft shadows which are produced by objects occluding each other, i.e. in creases and corners, or when two objects are very close together. It increases spatial awareness as it is easier to tell how close two objects are from how much they occlude each other.

With regards to computer games, Ambient Occlusion is a relatively new technique, with the original method actually being developed for use in CGI films. It was developed in 2002 by ILM (Industrial Light and Magic) and used ray tracing to calculate occlusion factors, i.e. “For every surface point, rays are cast in a hemisphere around the surface normal. The final occlusion amount is dependent on the number of rays that hit other surfaces or objects in the scene.” [Landis, 2002] This method was not used for real time applications but with non real time renderers for Pixar animated movies.

In 2007, the games developer Crytek devised a technique that could be used successfully in computer games. Screen Space Ambient Occlusion (SSAO) was first used in the computer game Crysis, and has since appeared in many other games. The technique is performed in screen (image) space and uses the depth buffer as input. It is implemented as a pixel shader and is executed solely using the GPU. SSAO is well suited for use in games and real time applications because, since it is performed in screen space, its performance is not dependant on scene complexity, making it faster than other methods in complex, detailed scenes. Of course, since it is a screen space calculation, it is not as accurate as geometry dependant ambient occlusion, and only approximates the effect. However, when it comes to computer games, physical correctness is not essential, rather it is the illusion of reality that is important. [Briney et al, 2010]

It wasn’t until 2009 that Crytek released a full description of their algorithm, and as such there are a number of different interpretations of the technique in use. The games developer Blizzard used their version of SSAO in the game StarCraft II (2010) and Nvidia also have their own version of the technique, called Image Space Horizon Based Ambient Occlusion (HBAO). Some implementations focus on improving the performance; while others are far better quality and some attempt to combine the two. Thus, it is clear that Screen Space Ambient Occlusion is a graphical method with room for multiple interpretations and expansion. With that in mind, the focus of this project has been to answer the following research question:

     

How far can real-time Screen-Space Ambient Occlusion methods be improved/expanded to better approximate Global Illumination while still remaining cost effective enough for games?

There are two facets to this question: One, an investigation of methods that will improve the graphical quality of the SSAO effect and therefore better emulate indirect illumination; And two, investigating ways in which the performance cost may be reduced, thus keeping the derived method viable for computer games.

Progress Report ~6

Application: By the end of this week I am aiming to have the practical element of the project pretty much finished. I cannot think of many more ways to improve my current SSAO/SSRad, without radically changing a lot of elements, so, though I will admit that my application is nowhere near perfect, I have started to tie up the loose ends and clean up my code. Perfectionist that I am, it's hard for me to admit that there isn't really such a thing as perfection anyway. After the application is tidied up, I will start taking frame time and frame rate readings for my results section.

Dissertation: I'm happy to report that I have made some progress with the dissertation. I have finished the introduction, and written one section of my literature review (albeit the shortest section). Tomorrow I shall hopefully start on the SSDO lit review and on the Methodology section.

Here are some screenshots of my SSAO/SSRad in action, and the buffers used in each step:


Depth/Normals buffer( size of screen):

Colour buffer ( 1/4 screen size):

SSAO buffer ( 1/2 screen size):

Blurred SSAO buffer ( 1/2 screen size):

          Low cost smart blur:

            Simple Gaussian blur:

Rendered scene without SSAO (just textured):

Rendered scene with SSAO:

Progress Report ~5

• Almost finished writing the dissertation introduction.
• I have managed to implement an edge detecting blur. It is not a completely accurate implementation, mathswise, but it is low cost. 
• Have started working on the global illumination aspect - I have looked at screen space radiosity techniques and implemented a simple local radiosity effect in screen space. This is essentially using the same ssao calculation but also taking into account scene colours (which are rendered into a half screen size texture).
• Next I plan to go over the SSDO paper in detail, both for my lit review and to develop more ideas for the GI aspect.

Progress Report ~4

Met with Matt on Wednesday to update him on my progress, and discuss what I plan to do next.

I have optimised the SSAO performance by using  downscaled SSAO and Blur textures. Both texture are half the screen size, and the final term is gained by upsampling the final blur texture using bilinear interpolation. This resulted in a considerable performance gain - the framerate(fps) is almost doubled, but the quality of the SSAO has suffered somewhat. However, I consider this to be an acceptable tradeoff, considering how much lower the frametime (ms) is now.

There are a few main areas that can be optimised to increase performance:

1. Texture Lookups: This is probably the most performance consuming aspect of the SSAO shader. Downsampling the SSAO and blur textures reduces the number of lookups and increases performance.
2. Shader Organisation: Combining shader operations and reducing the amount of code can help to optimise the shaders. I will probably leave this optimisation till the end, as it will reduce the readability of the code. 
3. No. of Samples: Reducing the number of samples does result in faster performance, however the quality dip is not acceptable. Currently the sample no. stands at 16, which provides a reasonable balance between quality and speed. 
4. Blurring: The type and range of blur used will also be a factor in the performance of the program. Right now, a very simple gaussian blur has been implemented which uses two passes - vertical and horizental. Other types of blur such as bilateral(edge detecting) have yet to be tested. 

In regards to the upsampling, I am currently looking into bilateral upsampling - which is essentially bilinear upsampling, which uses the normals and depths as additional weights in order to respect the edges. Thus, the edges of objects will not be blurred. I am concerned that this may decrease performance, as it requires more texture lookups, however I am going to try and implement it for comparison.

In regards to the blur pass, I am looking into implementing a smart blur which respects the edges - using normals/depth as a weight. As with the bilateral upsampling, this may impact the performance, but, depending on the quality increase, may be worthwhile.  

I hope to get the majority of this done before next week, and then I will start on the global illumination aspect of my shader.

Also - In my previous post, I stated that I had changed the way I calculated the depthDifference from my original implementation: ie. from OccluderSample.a - linearDepth to sample.z - OccluderSample.a, because the original calculation produced odd results. I have now realised, that this because I had them in the wrong order. It should have been: linearDepth - OccluderSample.a. This produces correct looking results - which to my eyes seem slightly more defined than the sample.z - OccluderSample.a results. For now, I have implemented the ability to switch between the two within the program.

Progress Report ~3

So, my last post detailed the current list of tasks for improving my SSAO shader and I'm pleased to say that I have made some good progress there.

Implement blur shader - test both Gaussian (using scene normals) and bilateral blurs:  I have implemented a very simple bilateral blur, which produces relatively good results. I have still to implement a gaussian blur using the scene normals/depth. 

Try and get rid of SSAO artifacts (as seen in previous post): As the screenshots to follow will show, I managed to get rid of the banding artifacts that were present in the SSAO term.At first, I thought it may be a depth precision issue, so I changed the texture format from D3DFMT_A16B16G16R16F to D3DFMT_A32B32G32R32F. This did get rid of the banding somewhat, but not completely. I finally figured out that it was a self occlusion artifact, and once I started using the dot product of the normals as a weight, the artifacts disappeared completely and the scene now looks a lot cleaner.  

Optimisation - using scene normals in SSAO calculation: The SSAO shader now uses the scene normals to both alter the direction of the ray and as a weight in calculating the AO term. This prevents self occlusion and improves the ao quality. 

Optimisation - try out different occlusion functions: The original ao calculation I was using was this; 

step(falloff,depthDifference)*(1.0-smoothstep(falloff,strength,depthDifference)); 

which produced the ao seen in the previous post's screenshots. I then added in the normals to the calculation as so;

step(falloff,depthDifference)*normDiff*(1.0-smoothstep(falloff,strength,depthDifference)); 

While this eliminated the self occlusion, it also produced ambient occlusion that didn't look quite right. I found that eliminating the step function from the calculation greatly improved the results, i.e.                                                 

     

if (depthDifference>= falloff && depthDifference <= strength)

finalColour += normDiff*(1.0-smoothstep(falloff,strength,depthDifference));  

   

However, I discovered that the reason that the previous ao calculation was wrong due to the way I got depthDifference. Originally I had been getting depthDifference from sampleDepth - linearDepth, i.e. the depth of the occluder pixel - the depth of the current pixel, both taken from the depth texture. Changing this to sample.z - sampleDepth (as done in the StarCraft SSAO), fixed the problem and I was able to go back to using:    

                          

step(falloff,depthDifference)*normDiff*(1.0-smoothstep(falloff,strength,depthDifference));

Optimisation - investigate reducing texture lookups: Reducing the size of any of the textures involved - i.e. Depth buffer, SSAO buffer, or blurred SSAO buffer - greatly improves performance, but also decreases graphical quality. I have come across a technique called bilateral upsampling which may help to remedy this, and I will be researching into it this weekend.

Optimisation -investigate other techniques that improve performance: I have been trying a number of tweaks to my shader: such as combining multiple operations in one, and decreasing the number of samples. However, the number of texture lookups is the main performance factor, I believe, and so that is my focus.

When it comes to graphical quality, theSSAO produced is quite noisy, and even blurring does not get rid of the noise completely. I am pretty sure that this is due to the way I am getting the random normals and is something I need to look into fixing, hopefully before Wednesday.

In regards to the dissertation, I am in the midst of writing the introduction and will try to start the lit review next week at some point. 

SSAO term:

Blurred SSAO term: 

Progress Report ~2

Another update! And so soon after the last one.

Met with Matt today, to update him on my progress and discuss what I'll be focusing on next.

There are two aspects to my research question - optimisation and global illumination. For the next fortnight I will be focusing on optimising the SSAO shader, since currently it's hitting my framerate quite substantially. Then I will focus on the global illumination aspect.

I left room for manouvering in my Gantt chart, so I should be okay for time - getting a cold and then the flu didn't help, but I'm confident that I can get everything/or most things that I planned to do, done.

Tasks for the next two weeks:

• Implement blur shader - test both Gaussian (using scene normals) and bilateral blurs
• Try and get rid of SSAO artifacts (as seen in previous post)
• Optimisation - using scene normals in SSAO calculation
• Optimisation - try out different occlusion functions
• Optimisation - investigate reducing texture lookups
• Optimisation -investigate other techniques that improve performance

I'm also going to start writing the literature review, this week and next week and I've made a list of possible topics to include:

• Background on Ambient Occlusion, and non screen space techniques
• Screen Space Ambient Occlusion - Crytek's method, and variations on it. e.g. Nvidia's Image Space Horizon Based AO, and Blizzard's implementation of SSAO in Starcraft 2.  
• Screen Space Directional Occlusion - from the paper "Approximating Dynamic Global Illumination in Image Space"
• Screen Space Radiosity - from an article in Shaderx7 - it's a very small section of the article but I'm trying to find more on it. 

I will also write up my draft introduction this week.

Progress Report ~1

So, it's been a while since my last update and while there has been progress, it has been slower than I would have hoped.

One thing that hampered the work flow was the flu, which resulted in me having to go home for week 4 to recover. Incidently, week 4 was supposed to be a technical supervisor meeting week, but that had to be delayed to week 5 instead.

However, the main thing that slowed progress, was the many problems that I encountered trying to implement SSAO in my d3d9 application. The shader I was using came from an article on Blizzard's implemention of SSAO in StarCraft 2 in the book Game Programming Gems 8 - there were a lot of variables that weren't really explained in the shader and I had to guess at a lot of the values I plugged into them. When I did get the shader to actually produce a result, the graphical effects it produced were obviously not ambient occlusion:

Some problems were easily solved (though easy to overlook as well), such as the effect of being able to see the SSAO effect of the room through the walls - this was caused by the Alpha Blending being turned on in the main file, so easily fixed. But even after fixing that, the scene still looked wrong. I am reasonably sure no that the main problem lay in the way the shader converted screen space positions to view space positions using the depth values, but at the time I could not work out why it was not working.

Suffice it to say, I spent a few weeks trying to get that shader to work, but ultimately I decided that it would be better to start a new SSAO shader from scratch and test each element step by step - which is really what I should have done in the first place. After spending weeks trying to get the GPGems8 shader to work and failing, it took me far less time than that to produce a new shader from scratch and get it working. However, I probably wouldn't have been able to get it to work so quickly if I hadn't been learning from those early failures.

I will detail the steps I took in implementing the new shader in another post, but for now here are some screens of the working depth texture and SSAO texture.

Depth and Normals (normals in rgb, depth in a):

 SSAO without blur:

 Unfortunately, there are some weird artifacts in the SSAO, that occur when the camera is at a certain angle to that back wall of the scene - I have yet to determine the cause:

26th January - Meeting with Technical Supervisor

So, today I had my first meeting of the semester with Matt Bett. It was pretty productive, though my brain is currently in the thrall of a nasty cold, and I gained a new sense of focus.

These past two or so weeks have been rather unproductive I must admit, mainly due to being ill and to me panicking slightly when I should have been focusing. My original plan of action for the project was to first create a demo application - containing lights, shadows, physics etc, which I could then implement the ambient occlusion into. But I think I was placing  too much importance on that aspect, and my talk with Matt made me realise that I should focus my efforts on the Ambient Occlusion, on actually anwering my research question and not on having an amazing 3d scene. If I have time at the end of the project, then I may go back and improve my framework but all I really need is a reasonably simple directx framework with the ability to use models and shaders.

I already have a basic framework to work with - I stripped back the framework I used for the maths coursework - so my tasks for the next fortnight is to get a simple 3d scene running with the basic SSAO shader. If I can get this done before the fortnight is over, then I will next think about the blending (or blur) that will be applied to the SSAO to make it look softer/less noisy.

Matt already suggested to different blending methods during the meeting - Gaussian blur (which is quite expensive) and a simple Bilinear blur (vertical blur then horizental blur) and I'm sure that some research will turn up even more variations.

Onwards!

Honours Project Proposal (Final)

The project proposal is all done and handed in, and the real work looms ahead. Oh dear.

You can read my Proposal here.