Problem statement: Black edges on object with transparency in Maya 2015, VRay 3.0.
Everyone who ever worked on integration of computer generated (CG) image over a life action video knows unpremult/premult workflow. The main ruke is before applying any color corrections to CG render it must be divided by alpha channel (unmultiplied). The whole thing about pre-multiplication and transparency is quite simple.
What in Nuke jargon sounds like “premult” in normal math language is
newRGB = RGB * Alpha
Similar for “unpremul”
newRGB = RGB / Alpha
The problem occur when you get footage which is already premultiplied and you multiply your RGB channel by Alfa again. In fact that exactly what happening when you try to use some footage with Alfa channel as transparency in Maya.
If you think your image is Unpremultiplied (but it’s really Premultiplied) and you add a Premult node you basically multiply the image twice (RGB x Alpha x Alpha). If you have an RGB pixel of 0.6 and you multiply it by the alpha pixel of 0.4 you get a correct value of 0.24 for that RGB pixel. But if you premultiply the image twice you are effectively doing 0.6 x 0.4 x 0.4 giving you a value of 0.096, which darkens the edge.
There are situations when you will want to Unpremultiply an image. The general rule for any form of colour correction is: Unpremult the image first, do the colour correction and then Premult back to its correct state. This is so you don’t accidentally colour correct any of the transparent Alpha edges.