I haven't posted in a bit, since everything is still a mess. I've also been doing experiments with projects I can't share so that's another reason. But I thought I'd catch up on what I've been trying out so far. What has or hasn't worked.
So I'm currently trying to devise a workflow that allows me to bring scenes from 3dsmax, with all the glorious lighting from V-Ray baked into them.
Here's the current method:
1. Optimize all geometry as much as possible. Reducing turbosmooths, reducing curve interpolation on rendered splines and/or sweeps. Some things I've just applied an optimize modifier to reduce.
2. Unwrap. I usually do a shoddy job of just putting simple box mapping on objects. So before I start combining meshes, I unwrap them as properly as possible. I'm still trying to find a better workflow that doesn't damage the existing mapping until after baking.
3. Combine. At this point I combine the whole entire scene into a single object. This is simply so that I can explode it back into pieces based on material. This is because if I have two of the same material in the scene and bake them as two textures - the material is confused as to which baked texture to use.
detach by material name
4. Organize. Now each mesh should have a jumble of UVs, so it needs to be packed more cleanly. Currently I'm simply using the 3dsmax pack uv function, but I'm looking into other plugins to do this better.
5. Bake! Everything should be ready to go. A couple important things are that; you make sure you're using regular cameras to preview the lighting. You can still use a vray exposure control to match an exposure so that it works with Render To Texture. Also make sure you're baking your gamma into the render. For me this is simply changing my color mapping mode to 'color mapping & gamma'. Also I have the baked texture replace the existing diffuse slot texture, so that I can convert the material to standard and export with the geometry without issue.
In the Render to Texture panel, check 'color mapping' & select the desired elements to bake:
VrayCompleteMap - everything
VrayDiffuseFilterMap - just diffuse
VrayRawLightingMap - just lighting/shadows
VrayMtlReflectHighlightGlossinessBake - just the texture used in the reflection
VrayBumpNormalsMap - normals from bump material (if any)
I've gotten a lot of issues during this step - ranging from issues if any of the materials are anything besides a regular VRayMtl. So blend materials, 2-sided, light material will cause issue, the issue being that render to texture can't find the diffuse slot of that material to place the baked texture into.
6. Export. Once everything has baked. If you've set the baked materials to load into the diffuse slots. Next you'll convert all the materials to standard. I use this:
At this point I also use a script to load the diffuse textures into the viewport so they're visible. I can't find the exact script but there are multiple ones on scriptspot.
Now you can export everything as an FBX, I embed media, and for units scale I uncheck automatic and set it to centimeters.
7. Import. The FBX should load automatically if you exported it into your Assets folder somewhere. The model should also have all the baked textures on all its materials. You can now go through them and change them to more complex materials and load the other baked materials if needed (bump & reflection). I'm still struggling to find the best methods for reflections since all my materials usually have some sort of subtle reflective property w/ fresnel. I'll probably wait for Unity 5 before I purchase anymore assets - but I have been eyeing something like Candela SSRR or Screen Space Reflections (SSR). I still need to figure out and make use of the Shader Forge asset I purchased :)