The cancel button for a form is the button control that is clicked whenever the user presses the ESC key. The button assigned to this property must be an IButtonControl that is on the current form or located within a container on the current form.
This property allows you to designate a default action to occur when the user presses the ESC key in your application. You can use this property to allow the user to quickly navigate a simple form by allowing them to simply press the ESC key to close a window without committing changes instead of manually clicking the cancel button with their mouse.
this.CancelButton = buttonCancel;
Cancel Button in Windows Forms (C#)
For setting the same in VBA/VB refer : http://vbadud.blogspot.com/2007/06/setting-default-cancel-buttons-in.html
See also Setting Default & Cancel Buttons in VBA/Visual Basic