Bumpmapped low-poly tree volumes technique (rant)
Posted: (080207)(15:02.47)
I decided to post my little document about creating normalmaps for foliage using Doom3 technology. Originally I was only going to share this with a few people within the community but decided it's not a big deal and there are many that could benefit from this information.
Here's the original thread at Doom3world.org:-
http://www.doom3world.org/phpbb2/viewtopic.php?t=21149
And here below is the doc itself:-
__________________________________________________
__________________________________________________
I've been getting back to doing some work on tree models but as we all know it's difficult because trees are volumes and you need to have a lot of surfaces intersecting to do a good low-poly tree, and the more low-poly, the more trees you can have in one spot, which is also what we want. The problem is that rendering a flat surface in the likes of Doom3 is going to be precisely lit and never match the light of intersecting surfaces on different angles. So we tend to see trees in Doom3 baring very harsh seams along intersections, or without a lot of volume to minimise intersections, or dull flat lighting using strong ambient light, or using single-pass blends and avoiding alphatest altogether. I've tried them all, none are ideal.
I was looking at some real trees and realised they're like volumes of individually faceted leaves on all angles, always appearing bright and dark mixed together, and I thought this can't really be done with a normalmap, the problem is using normals that only project from the limits of its parent surface, so in a "traditional" normalmap bake no matter how many different angles you put your leaves on, it's still basically lit relative to its flat parent surface and any surface intersecting will appear to be lit very different and therefore creating harsh ugly noticeable seams.
Then I remembered some normalmaps I borked a while back that somehow captured light incorrectly, I got some yellow in the normalmap which caught light from the opposite direction, and looking at the tree I realised this could really work well within a single normalmap of a leaves material for a tree model, by truly having each individual leaf on an entirely different angle to the point where it would capture light from a light source behind the geometry itself, which sounds impossible but it really can happen. Ideally you never want to see yellow in your normalmaps but for normalmapped alphatest materials trying to create volumes, this could be the answer. It would mean no matter where strong light is coming from, there would be a balance of light and colour, at least some of the leaves on the surface will be catching it and give the illusion that the surface isn't just a flat twosided material but truly more like a volume, and it would equalise the lighting on all surfaces, leaf by individual leaf as like the true nature of a tree, instead of flattening the light on the entire surface affecting every leaf. Cynically you might say it's making such a horrible mess of the normalmap it doesn't know what the hell it's doing, but you can have low-poly trees capable of great light interactions this way without noticeable seams or any other gimmicks to hide them.
[ external image ]
Here we have a quick test to demonstrate inverted normalmaps. I got this normalmap by actually properly baking it using a negative projection depth, it is a proper bake, so it is a true negative and not just inverted colours of the normalmap image, even though that produces _exactly_ the same result and that's how I will be mostly doing it later. The blue circle catches light from the proper direction, but the green circle is inverted and catching light from the opposite direction of the actual light source, as if the light is in a mirrored position relative to the surface but only for those specific texels. There is only one light in that scene and you can clearly see it's on the other side of the geometry. Apart from that, it's still dynamically lit as the light source is moved around like a regular normalmap would behave, and still relative to the light source in other ways. It's still a valid normalmap and using valid normalmap colours, just producing a result flipped around to the opposite side. Never really purposely used, but still valid as in it doesn't cause any performance issues or cause Doom3 to crash or anything like that. To benefit it would also seem the green circle isn't actually catching an equal amount of light there even though it's on the exact same quad as the blue, see how the blue bleeds at specular but the green doesn't, even though it's using one material and one set of textures. Why on Earth would you want to do that though? Well now to actually applying this to some leaves material for a tree model.
[ external image ]
[ external image ]
By having the normals of individual leaves in all directions and not only within the limits of it parent surface you can start to create some true volumes that at least minimise harsh seams along intersecting geometry and under different lighting conditions. To create these reverse normals from the opposite side you need give the normalmap negative/inverted colours, then rotated 180? or flipped/mirrored, and you get your inverted normalmap that will catch light/cast shadows from the opposite direction. This is a million times easier than actually baking negative projections separately which I don't know if that's even possible using Doom3 renderbump. In theory it sounds like it's doing the job of the "twosided" keyword in a material but that doesn't solve lighting issues, it makes a material appear from both sides of the surface but still lights it from one.
The way I've done it in my leaves material is to split the hi-res leaves model into 2 parts, with the second less-populated part later to become inverted, which also will appear to rest behind (underneath) the main "traditional" layer. You want to have minimal number of inverted leaves but enough to separate the proper normalmap from itself and make the leaves appear to spread out and still have a balance of all variety of angles, even though the texture is quite occupied there's still plenty of transparency. Balance here is the key, keeping in mind how all of these leafy angles attribute to the normalmap work as a whole. I then create separate normalmaps from each layer, but I rotate the second (underneath) layer 180? before baking the normalmap, then I invert the colours and rotate back 180?. This ensures the angles are correct when inverting the colours because inverting a normalmap also assumes it's flipped and mirrored, which we don't want. Then I combine the two normalmaps, I used an alpha in the "traditional" layer and just filled the inverted normalmap back into it, to create one single, very colourful normalmap. You want to keep the traditional normalmap component layered over the inverted normalmap component, as well as combine it with some leaves on different angles for both the traditional front and the inverted back, so you're not just dealing with leaves that are either front or back, but mixed together using all angles. Because you're trying to create as much of a volume as you can, you can afford to have leaves on sideways angles in your material.
[ external image ]
[ external image ]
Here you see a quick and dirty final comparison between the inverted-combination normalmap and a "traditional" "flat" normalmap. It doesn't solve intersections entirely of course, but you can see, especially from a distance, how the main seams and the general lighting conditions of each intersecting surface is drastically minimised using inverted normalmaps in harmony with proper normalmaps. It especially pays off if you want to have strong dynamic lights. Using the traditional method it makes the actual model structure so apparent, but using this new technique it allows the materials to behave more like a volume and disguise the model structure while still allowing for plenty of light interaction, contrast and detail, not to mention actually having bumpmapped trees with specularity.
So, leaf by leaf you can tell which direction the main light is coming from, but as a whole flat surface it becomes difficult to tell because there's so many different angles on there, in front and behind, but then ultimately as an entire tree object, from a distance, I find you get the best of both aspects, it seems the entire material works as a whole to at least give the impression of a volume. As an entire tree, it enables the material to catch light mostly from the proper direction by the normalmap, but also all light from all directions as to avoid letting the geometry dictate what light and how much.
One major downside to this is that I don't like your chances of using photosource images to quickly work with, you're really going to need to get your hands dirty with some high-definition modelling to do this. It requires a careful balance of angles in the normalmap that you can't possibly do by converting images directly into normalmaps.
To finish, I want to mention some extra little techniques for spicing up alphatest, especially trees. They come at a price but they are pretty nice. One is that alphatest isn't texture-filtered nor can it be antialiased, so you notice as you move further away from it, it becomes these harsh and sharp floating lumps of nonsense pixels because it can't filter into anything easy on the eyes, only ons and offs. You can use a "fuzz" material which is basically white with a dark coloured silhouette outline (gl_zero , gl_one_minus_src_color) around every sharp alphatest edge in your material, as just a little light/fog-friendly helper to give your tree back that soft fuzzy feeling, especially when viewing at extra-close or extra-far distances. I can't see this as a problem because it's a single pass texture that doesn't use normalmapping and can't be mapped onto by lightshaders and such, so I wouldn't assume this as being a massive performance killer as an extra. Visually it's up to you how you think it looks and helps.
Next you probably all already know about vertex-painting/vertex-blending, well basically you blend your material with a different material via vertex-painting, or to be more efficient you can half-blend your material with basic single-pass fullblack, and give the foliage a little more overall depth using the vertices instead of trying to manage it all in one big normalmap. Again, I don't assume this as major performance draining extra, so why wouldn't you use vertex blending like that. Again with the material itself, you could go ahead and give it some subtle very low-res cubemap reflections, with all the chaos in the normalmap the cubemap reflections will spread out evenly and may give your tree a subtle dewy or wet appearance. This can be a performance drain but personally if it's done right a subtle reflection can have beautiful results, moreso in Quake IV where cubemap reflections don't glow-in-the-dark. Cubemapping does warp/flow with the normalmapping within the alphatest of course which is why it's considered for use. Also material-wise you can give it a subtle swaying movement using the "deform turbulent" keyword, although this doesn't work well with alphatest and you'll only get the alpha swaying/wobbling, you may still like to have that feature, although if you can this would be far better done as an animating scenery object. Lastly concerning materials, you may like to blend in some "detail mapping" to give the foliage more grittiness and perception of a higher-detail texture. Again, that's a personal choice but personally I'll probably find myself doing it as usual. It's debatable whether or not it's something that has a place with normalmaps in Doom3, but it's easy to turn off, and I like it, so I use it.
Next is casting proper shadow for your whole tree, and not just the trunk and branches, because as you know alphas can't cast shadows, only whole geometry. This involves a few techniques. First take a screenshot of your tree (and only your tree against a black background) from the exact position and angle of the light that is going to cast the shadow, but viewing it aligned with the middle vertical range of the tree and exactly horizontal, preferably orthographic. Get it fullbright from all angles so it's all white, and take a high-res screenshot. You can then create an alpha of how the whole tree looks from that angle, preferably only 2-colour (1-bit) fullblack and fullwhite image with no antialiasing. Next you want to use that image as a displacement, and displace a heavily subdivided plane so the tree shape sticks out, then you can easily delete the lower (background) parts of the plane so you now should have just a flat model of the silhouette of your tree shape, to some triangulated degree. Find your nearest optimisation tools and get to work at it until you've used enough judgement to get what will make a good shadow caster but without going insane with the polycount, taking into consideration what parts will be casting shadow onto what in-game, and with the help of the common/shadow material, align your shadow-caster to face the light source and this will be an invisible nonsolid but shadowing-caster for your tree, complete with the foliage shadow, while the actual tree model materials will cast no shadows at all. There are probably better/easier ways to get this done, but that's how I do it.
I was also thinking, with this technique it may just be possible to use some alphatest autosprites well enough now. That means intersections will be increased, but your tree/weed/bush will have better coverage/filling from ALL angles, so it will always look "full" from all angles with proper use of autosprites. Sometimes a cleverly-placed autosprite can fill in some gaps that would take more than a quad otherwise, plus I think if used appropriately it can make an interesting effect when you combine autosprites with static geometry, but it's up to the individual taste. You may find for creating a bush/hedge that clumps of autosprites may work better than a static mesh, but that's up to you. Definitely something I will experiment more with now.
Okay, I'm ready to shut up. I'll post back here when I actually put my money where my mouth is and create some proper big trees properly using these techniques and hopefully get a whole bunch of them together and looking and performing decently.
Here's the original thread at Doom3world.org:-
http://www.doom3world.org/phpbb2/viewtopic.php?t=21149
And here below is the doc itself:-
__________________________________________________
__________________________________________________
I've been getting back to doing some work on tree models but as we all know it's difficult because trees are volumes and you need to have a lot of surfaces intersecting to do a good low-poly tree, and the more low-poly, the more trees you can have in one spot, which is also what we want. The problem is that rendering a flat surface in the likes of Doom3 is going to be precisely lit and never match the light of intersecting surfaces on different angles. So we tend to see trees in Doom3 baring very harsh seams along intersections, or without a lot of volume to minimise intersections, or dull flat lighting using strong ambient light, or using single-pass blends and avoiding alphatest altogether. I've tried them all, none are ideal.
I was looking at some real trees and realised they're like volumes of individually faceted leaves on all angles, always appearing bright and dark mixed together, and I thought this can't really be done with a normalmap, the problem is using normals that only project from the limits of its parent surface, so in a "traditional" normalmap bake no matter how many different angles you put your leaves on, it's still basically lit relative to its flat parent surface and any surface intersecting will appear to be lit very different and therefore creating harsh ugly noticeable seams.
Then I remembered some normalmaps I borked a while back that somehow captured light incorrectly, I got some yellow in the normalmap which caught light from the opposite direction, and looking at the tree I realised this could really work well within a single normalmap of a leaves material for a tree model, by truly having each individual leaf on an entirely different angle to the point where it would capture light from a light source behind the geometry itself, which sounds impossible but it really can happen. Ideally you never want to see yellow in your normalmaps but for normalmapped alphatest materials trying to create volumes, this could be the answer. It would mean no matter where strong light is coming from, there would be a balance of light and colour, at least some of the leaves on the surface will be catching it and give the illusion that the surface isn't just a flat twosided material but truly more like a volume, and it would equalise the lighting on all surfaces, leaf by individual leaf as like the true nature of a tree, instead of flattening the light on the entire surface affecting every leaf. Cynically you might say it's making such a horrible mess of the normalmap it doesn't know what the hell it's doing, but you can have low-poly trees capable of great light interactions this way without noticeable seams or any other gimmicks to hide them.
[ external image ]
Here we have a quick test to demonstrate inverted normalmaps. I got this normalmap by actually properly baking it using a negative projection depth, it is a proper bake, so it is a true negative and not just inverted colours of the normalmap image, even though that produces _exactly_ the same result and that's how I will be mostly doing it later. The blue circle catches light from the proper direction, but the green circle is inverted and catching light from the opposite direction of the actual light source, as if the light is in a mirrored position relative to the surface but only for those specific texels. There is only one light in that scene and you can clearly see it's on the other side of the geometry. Apart from that, it's still dynamically lit as the light source is moved around like a regular normalmap would behave, and still relative to the light source in other ways. It's still a valid normalmap and using valid normalmap colours, just producing a result flipped around to the opposite side. Never really purposely used, but still valid as in it doesn't cause any performance issues or cause Doom3 to crash or anything like that. To benefit it would also seem the green circle isn't actually catching an equal amount of light there even though it's on the exact same quad as the blue, see how the blue bleeds at specular but the green doesn't, even though it's using one material and one set of textures. Why on Earth would you want to do that though? Well now to actually applying this to some leaves material for a tree model.
[ external image ]
[ external image ]
By having the normals of individual leaves in all directions and not only within the limits of it parent surface you can start to create some true volumes that at least minimise harsh seams along intersecting geometry and under different lighting conditions. To create these reverse normals from the opposite side you need give the normalmap negative/inverted colours, then rotated 180? or flipped/mirrored, and you get your inverted normalmap that will catch light/cast shadows from the opposite direction. This is a million times easier than actually baking negative projections separately which I don't know if that's even possible using Doom3 renderbump. In theory it sounds like it's doing the job of the "twosided" keyword in a material but that doesn't solve lighting issues, it makes a material appear from both sides of the surface but still lights it from one.
The way I've done it in my leaves material is to split the hi-res leaves model into 2 parts, with the second less-populated part later to become inverted, which also will appear to rest behind (underneath) the main "traditional" layer. You want to have minimal number of inverted leaves but enough to separate the proper normalmap from itself and make the leaves appear to spread out and still have a balance of all variety of angles, even though the texture is quite occupied there's still plenty of transparency. Balance here is the key, keeping in mind how all of these leafy angles attribute to the normalmap work as a whole. I then create separate normalmaps from each layer, but I rotate the second (underneath) layer 180? before baking the normalmap, then I invert the colours and rotate back 180?. This ensures the angles are correct when inverting the colours because inverting a normalmap also assumes it's flipped and mirrored, which we don't want. Then I combine the two normalmaps, I used an alpha in the "traditional" layer and just filled the inverted normalmap back into it, to create one single, very colourful normalmap. You want to keep the traditional normalmap component layered over the inverted normalmap component, as well as combine it with some leaves on different angles for both the traditional front and the inverted back, so you're not just dealing with leaves that are either front or back, but mixed together using all angles. Because you're trying to create as much of a volume as you can, you can afford to have leaves on sideways angles in your material.
[ external image ]
[ external image ]
Here you see a quick and dirty final comparison between the inverted-combination normalmap and a "traditional" "flat" normalmap. It doesn't solve intersections entirely of course, but you can see, especially from a distance, how the main seams and the general lighting conditions of each intersecting surface is drastically minimised using inverted normalmaps in harmony with proper normalmaps. It especially pays off if you want to have strong dynamic lights. Using the traditional method it makes the actual model structure so apparent, but using this new technique it allows the materials to behave more like a volume and disguise the model structure while still allowing for plenty of light interaction, contrast and detail, not to mention actually having bumpmapped trees with specularity.
So, leaf by leaf you can tell which direction the main light is coming from, but as a whole flat surface it becomes difficult to tell because there's so many different angles on there, in front and behind, but then ultimately as an entire tree object, from a distance, I find you get the best of both aspects, it seems the entire material works as a whole to at least give the impression of a volume. As an entire tree, it enables the material to catch light mostly from the proper direction by the normalmap, but also all light from all directions as to avoid letting the geometry dictate what light and how much.
One major downside to this is that I don't like your chances of using photosource images to quickly work with, you're really going to need to get your hands dirty with some high-definition modelling to do this. It requires a careful balance of angles in the normalmap that you can't possibly do by converting images directly into normalmaps.
To finish, I want to mention some extra little techniques for spicing up alphatest, especially trees. They come at a price but they are pretty nice. One is that alphatest isn't texture-filtered nor can it be antialiased, so you notice as you move further away from it, it becomes these harsh and sharp floating lumps of nonsense pixels because it can't filter into anything easy on the eyes, only ons and offs. You can use a "fuzz" material which is basically white with a dark coloured silhouette outline (gl_zero , gl_one_minus_src_color) around every sharp alphatest edge in your material, as just a little light/fog-friendly helper to give your tree back that soft fuzzy feeling, especially when viewing at extra-close or extra-far distances. I can't see this as a problem because it's a single pass texture that doesn't use normalmapping and can't be mapped onto by lightshaders and such, so I wouldn't assume this as being a massive performance killer as an extra. Visually it's up to you how you think it looks and helps.
Next you probably all already know about vertex-painting/vertex-blending, well basically you blend your material with a different material via vertex-painting, or to be more efficient you can half-blend your material with basic single-pass fullblack, and give the foliage a little more overall depth using the vertices instead of trying to manage it all in one big normalmap. Again, I don't assume this as major performance draining extra, so why wouldn't you use vertex blending like that. Again with the material itself, you could go ahead and give it some subtle very low-res cubemap reflections, with all the chaos in the normalmap the cubemap reflections will spread out evenly and may give your tree a subtle dewy or wet appearance. This can be a performance drain but personally if it's done right a subtle reflection can have beautiful results, moreso in Quake IV where cubemap reflections don't glow-in-the-dark. Cubemapping does warp/flow with the normalmapping within the alphatest of course which is why it's considered for use. Also material-wise you can give it a subtle swaying movement using the "deform turbulent" keyword, although this doesn't work well with alphatest and you'll only get the alpha swaying/wobbling, you may still like to have that feature, although if you can this would be far better done as an animating scenery object. Lastly concerning materials, you may like to blend in some "detail mapping" to give the foliage more grittiness and perception of a higher-detail texture. Again, that's a personal choice but personally I'll probably find myself doing it as usual. It's debatable whether or not it's something that has a place with normalmaps in Doom3, but it's easy to turn off, and I like it, so I use it.
Next is casting proper shadow for your whole tree, and not just the trunk and branches, because as you know alphas can't cast shadows, only whole geometry. This involves a few techniques. First take a screenshot of your tree (and only your tree against a black background) from the exact position and angle of the light that is going to cast the shadow, but viewing it aligned with the middle vertical range of the tree and exactly horizontal, preferably orthographic. Get it fullbright from all angles so it's all white, and take a high-res screenshot. You can then create an alpha of how the whole tree looks from that angle, preferably only 2-colour (1-bit) fullblack and fullwhite image with no antialiasing. Next you want to use that image as a displacement, and displace a heavily subdivided plane so the tree shape sticks out, then you can easily delete the lower (background) parts of the plane so you now should have just a flat model of the silhouette of your tree shape, to some triangulated degree. Find your nearest optimisation tools and get to work at it until you've used enough judgement to get what will make a good shadow caster but without going insane with the polycount, taking into consideration what parts will be casting shadow onto what in-game, and with the help of the common/shadow material, align your shadow-caster to face the light source and this will be an invisible nonsolid but shadowing-caster for your tree, complete with the foliage shadow, while the actual tree model materials will cast no shadows at all. There are probably better/easier ways to get this done, but that's how I do it.
I was also thinking, with this technique it may just be possible to use some alphatest autosprites well enough now. That means intersections will be increased, but your tree/weed/bush will have better coverage/filling from ALL angles, so it will always look "full" from all angles with proper use of autosprites. Sometimes a cleverly-placed autosprite can fill in some gaps that would take more than a quad otherwise, plus I think if used appropriately it can make an interesting effect when you combine autosprites with static geometry, but it's up to the individual taste. You may find for creating a bush/hedge that clumps of autosprites may work better than a static mesh, but that's up to you. Definitely something I will experiment more with now.
Okay, I'm ready to shut up. I'll post back here when I actually put my money where my mouth is and create some proper big trees properly using these techniques and hopefully get a whole bunch of them together and looking and performing decently.
![[ external image ]](http://www.violationentertainment.com/temp/SHleafusbush080206a.jpg){kind=link}
![[ external image ]](http://www.violationentertainment.com/temp/SHleafusbush080206b.jpg){kind=link}
![[ external image ]](http://www.violationentertainment.com/temp/SHleafusbush080206c.jpg){kind=link}
![[ external image ]](http://www.violationentertainment.com/temp/SHleafusbush080206d.jpg){kind=link}
![[ external image ]](http://www.violationentertainment.com/temp/SHleafusbush080206e.jpg){kind=link}