This section will show how to properly set up a baking set with Mental Ray, and V-Ray as well as the global settings that apply to both.
Using Raytrace shadows will produce the best shadows for baking. In the lightShape1 properties, drop down the shadow box and turn on Use Ray Trace Shadows. In this box you will also see Light Angle, Shadow Rays, and Ray Depth Limit.
Light Angle controls the sharpness of the shadow. Shadow Rays control the edge quality of the shadow, less rays mean a grainier edge. And Ray Depth Limit controls the amount of times a ray can be reflected before losing ability to cast shadow.
Under Window > Settings/Preferences > Plug-in Manager, ensure that the vrayformaya.mll plug-in is loaded.
Next enable the Rendering menu set, drop down menu under File.
While editing the lights to achieve the desired look you can actively see your lighting update using the IPR setting in the Render View. (Note: The max resolution for IPR is 600×450 so anything larger will not render.) Another thing to note, is the baked texture will not look as it does in the IPR. IPR should be used only to set up light direction, it uses dimmed settings so the actual light will look blown out once the baking is done. Use the true render from V-Ray to tweak lighting settings.
Now open the render settings under Window > Rendering Editors > Render Settings or the icon in the toolbar, and use these settings as a base.
Next select the objects you wish to add to the V-Ray baking. Go to Lighting/Shading > Assign single V-Ray bake options to selection, and change settings to match this as a base.
The image dimensions are not maxed at 2048px, they can be whatever you wish, just type in what dimensions you would like. To add more objects to this V-Ray bake set, simply select the objects and go to Lighting/Shading > Assign existing V-Ray bake options to selection and select which options you would like.
To avoid black edge bleeding, you will want to raise the Edge padding slider in the bake options window. The amount depends on the physical size of the objects. Bleeding will occur anywhere that UVs seams are broken.
Now to bake go to Lighting/Shading > Bake with V-Ray… and open the options box at the right side of the menu. Enter your output file location and choose Bake or Bake and close. (Note: Baking will not work if you currently have Render View open)
Mental Ray Setup
Under Window > Settings/Preferences > Plug-in Manager make sure the mayatomr.mll is loaded.
Change the menu set to Rendering, drop down menu under File.
Now open the Mental Ray Baking Options, Lighting/Shading > Batch Bake (mental ray), choose the option box to the right, and use these settings as a base.
After setting all options click Convert and Close or Convert. The files will bake to the project folder, and Maya will also give you the location of the last texture baked on the command window at the bottom right of the window.