Mod Lordgit's back with another look into the deep workings of NXT - this time, focusing on what exciting developments might lie in its future.
Today - following on from my previous blog - Ill cover some potential future developments for NXT, and the challenges that await us in making them a reality.
The level of technical detail will be somewhat akin to the previous blog, but due to the number of topics covered, I wont be going into quite as much devilish detail this time around. Also, I must point out that none of these future developments are set in stone yet.
Once weve got everyone up and running on the new client, then well be able to think about which of these we can bring to RuneScape.
With that said, let us begin!
Normal Maps and Increased Texture Resolutions
Probably at the top of our features wish list would have to be normal maps and higher-resolution textures. These two improvements together would give the most visual bang for buck after NXT is launched.
Normal maps would give the appearance of more geometric detail in the form of surface normals, to objects currently look very flat when lit. This would have the knock-on effect of bringing out much more lighting detail, especially with the new lighting features that take surface normals as inputs (i.e. irradiance, ambient occlusion, improved specular, etc).
However, there are two major challenges facing us when it comes to normal maps. The first is the massive undertaking of actually producing the from high-res versions of all the existing models in RuneScape, many of which dont even exist! The second is figuring out how to carefully introduce them so to not destroy the visual continuity of the game. Theres also the issue of increased GPU memory, but this should only affect players with very old machines.
The second of the texturing improvements comes from permitting artists to create higher-resolution textures. For the most part, artists are currently restricted to creating textures of resolution 128x128 resolution, which is extremely low for a modern game, especially as some games now support 4K textures! Even doubling the resolution to 256x256 would give us lots more texture surface detail to play with, but personally Id like to see us go even higher. Again, the challenge here is the same as with normal maps: production of high-res textures, increased GPU memory requirements and continuity across the game world.
Improved Animation System
For some time now, a new animation production pipeline has been in the works. This is going to allow us to finally use industry standard tools like Maya to produce our animations. It will remove our reliance on our in-house animation tools and pipeline, which is esoteric, cumbersome to work with and produces a very inefficient data set. The new pipeline will allow for animations with a proper joint system, but the big challenge has been in automating the conversion of all our existing animations over to the new format, a truly epic feat!
These tool changes will drastically improve the quality of life for our animators, as animations are currently very time-consuming and awkward to produce. Further pipeline improvements are also required to enhance the fidelity of RuneScapes animations.
One such improvement is moving over to a pure quaternion-based animation pipeline that doesnt massively quantise the frame data during data export, and keeps everything floating point-based during the entire export pipeline starting from Maya. This will help to eliminate fizzing artefacts and also make animations more fluid and efficient to produce.
Point Light Shadows and Percentage Closer Soft Shadows
To help complete our unified dynamic lighting and shadowing system, the addition of point light shadows is fundamental! Ive already talked about how challenging it was to get our fully dynamic sunlight shadow system running well while looking good, and it will be no different for point lights. In fact, it may be even more difficult as were going to have to support shadows from many light sources, so possibly moving over to a deferred shadowing system or tile-based rendering pipeline for high end (i.e. Forward+) might be required.
There are also improvements we can make to the existing shadow map filtering. Currently we use a series of box filter kernels when sampling from the shadow maps to give the illusion of soft edged shadows, but this is faking it at best. Percentage closer soft shadows attempt to accurately render contact hardening shadows that exhibit an umbra and penumbra region based on occluder distance, but this doesnt come cheap, so again it will be something just for high-end GPUs to chew through.
Improved Global Illumination
As you may have seen in my previous dev blog, the addition of irradiance and ambient occlusion have had a significant impact on the lighting in RuneScape. The question now is how can we improve on this?
The new GI system we have is a major improvement over the Java client, but cant really be improved on too much. We could use a combination of screen space directional occlusion (SSDO) and large-scale ambient occlusion, but the benefit would be minimal.
The best way forward is to truly take things to the next level. This would come by way of a real-time GI solution known as voxel cone tracing, which is essentially highly approximated ray-tracing to obtain single bounce indirect lighting from the scene. Currently the best implementation is provided by Nvidia GameWorks, but this relies on very high end GPU features, so not many players would have access to it.
There is a less computationally expensive solution known as cascaded light propagation volumes (LPV), which would work on more hardware, but quality and stability are a problem, especially over large distances. Some games that use LPV have a hybrid approach where they fall back to SSDO after the LPV has reached it limits, which could also be an option for us.
Improved Reflections and Image-Based Lighting
Reflections in games are actually a way of simulating indirect specular lighting from the environment, and in real-life, every object exhibits this to some degree. How much of the environment is reflected from an object depends on the shininess of that object's surface. Screen space reflections and parallax-corrected cubemaps are two modern techniques that attempt to get around the limitations of standard planar reflections and hand-placed environment probe reflections.
Probably the best we can hope for initially is screen space reflections on models that currently exhibit environment mapping, which would also allow for reflections on curved surfaces. Beyond that, it would be nice to combine this with parallax-corrected cubemaps, but that would require involvement from artists and pipeline changes.
Physically Based Shading
Physically based shading has been around for quite a while now in modern game rendering. Its a name given to the use of improved lighting equations that attempt to simulate how lighting interacts with object surfaces in a way that is more physically correct than other than traditional lighting models used by games. Its also a way of authoring materials and textures that work independently of the lighting environment. A lot of the techniques are actually just brought over from the movie visual effects industry, but have only become feasible with the arrival of modern-gen GPU hardware.
Another poor aspect of RuneScapes current lighting model is the general lack of specularity, or to put it another way, how un-shiny everything is. NXT now always applies a default amount of specular lighting to terrain and models if not specified by the artists, which has improved lighting vibrance to some degree. However, because its on everything, it needs to be subtle to avoid everything looking like shiny plastic.
PBS can already be achieved programmatically to some degree by simply adding an improved bidirectional reflectance distribution function (BRDF) to the lighting equations used for model and terrain rendering in the NXT client. Without artist involvement though, the full benefits would not be seen. If time permits, we may improve in the higher quality lighting settings, especially now that the new graphics options give us more flexibility in this regard.
Improved Light Attenuation/Falloff and Area Lights
The current point light attenuation and falloff model used by RuneScape is unintuitive for the artists, and a cause of many lighting artefacts seen by both Java, and NXT to some degree. Moving over to a standard point light attenuation/falloff model (or even a physically based inverse square falloff, as used by Unreal 4) would make light falloff more natural, and improve the lives of our artists. The problem in making this a reality, as with so many other improvements, is how to update the existing point lights without destroying the lighting environments already in existence.
Area lights (non-punctual light sources) would allow artists to have greater control over the areas that lights influence. Examples of these are lights with a flat rectangular surface. Area lights also come hand-in-hand with physically based shading. The challenge would be pipeline integration, engine support and training artists to use them effectively.
Improved Particle and Billboard Lighting
Another feature Im keen to improve is the lack of any lighting on particles and billboards. Realistic particle lighting is still not a well solved problem in real-time, but there are solutions which go beyond basic Lambert diffuse that try to take into account the density of particle clouds to simulate self-shadowing and translucency. Combining this with particle shadow casting would give a more consistent look to scenes and has the added benefit of being able to reuse the same particle effects in different lighting environments.
Volumetric Lighting and God Rays
Previous-gen games have modelled aspects of volumetric lighting like god rays using a single post process technique. As nice as this looks, its quite unconvincing, often exhibits lots of artefacts and is limited to just one dominant light source.
The best unified solution for simulating natural phenomena such as god rays is a technique known as volumetric light scattering. This is a more comprehensive solution that permits god rays from many light sources, not just the sun and also takes care of all your atmospheric scattering requirements at the same time. Neither of these techniques would require much involvement from art or pipeline, making them ideal candidates for improving our existing fog and light scattering model.
Vulkan and DirectX12 and Metal - oh my!
There are some new kids on the block when it comes to rendering APIs. These new APIs are lower level than current forms of OpenGL and DirectX, and are more console-like in their interfaces, all with the intended goal of reducing CPU overhead when making API calls, and enabling multi-core rendering.
It may come as a surprise when I say that none of these new APIs will give you much benefit by just a straight port from OpenGL or D3D, assuming all rendering is just running from a single thread. However, where they really shine is in allowing developers to build GPU command buffers for render passes on multiple CPU cores. But even then, if your applications is already GPU bound, overall performance gains wont be seen. On most machines that can run NXT, were predominantly GPU bound, especially on low-end hardware.
Thats not to say we dont intend on supporting one these bad boys! The truth is we cant wait to sink our teeth into at least one of them, anything that has a chance of increasing performance is like candy to us graphics developers. Out of the three, Vulkan will likely be the winner, given that it is cross platform, and adding additional backends for DirectX12 and Metal wont give us much extra, just take up more developer time and add additional complication to our rendering layer.
It wont simply be a case of just adding a Vulkan backend though, as Ive mentioned, multi-core rendering is where these APIs really fly, so some re-design of our rendering architecture will be needed to enable this.
Improved Multi-core Utilisation
The NXT client does make use of additional CPU cores for the purpose of asset loading, irradiance, disk cache operation and audio. This is why in-game loading waits have been substantially reduced for most people with a powerful CPU. However, this has all been done in a very ad-hoc way for specific systems, and for the most part, additional CPU cores are sitting idle, especially when nothing is loading.
So how do we tap into more of this idle CPU power you ask? What we need to develop going forward is a general purpose multi-core job system. With such a system, we can do more fine-grained parallel processing for systems that are suited to it. Two such examples are animation and particle updates, which can be very CPU intensive, and are still single threaded even in the NXT client.
If we do add support for Vulkan, then any new job system will play a key role in making multi-core parallel command buffer construction a reality. This would definitely ensure that many cores would be fully utilised during most of the frame, leading to massive gains in CPU performance.
Web Browser Support
Running the new client in the browser is still very much on the cards, but this will not be possible on launch due to the current limitations in browsers (i.e. lack of real threads, vector instruction support and WebGL 2.0). However, the good news is that these features are enabled to some degree in beta version of browsers, so at some point in the near future (hopefully late 2016) they will make it to public releases. At that point, we should also have a browser version of RuneScape NXT ready to go, as we already have beta access to these missing features, enabling us to continue development until they become publicly available.
A lot of you have asked if RuneScapes user interfaces are going to work correctly on 2, 4 and 8K displays in the future. The answer is that currently they wont work well on these high PPI displays because everything is placed according to exact pixel coordinates on screen.
Correcting this is a very big job, and the ideal solution of using a normalised device space to place everything might not actually be viable given the amount of interface components that have been bespoke scripted. We may have to think outside the box on this in order to devise a system that can intelligently scale the interfaces according to resolution.
We hope that NXT is just the beginning, and after launch we can finally crack on with these exciting new features to give you a much improved and more immersive RuneScape experience.
Thanks for reading!
Lead Graphics Programmer