The Step distance is the increment the cut or protrusion will move in when you visually drag it around with the cursor. For example, set 0.001", and it will look like it expands fluidly. Set 0.5", and depending on your model scale, your cut now expands in 'blocks' as you tug the cursor around.
You are in "Finite Extent" mode for the cutout, meaning it wants you to specify a distance for the extent of the cutout. There are other modes, such as "Through Next" and "Through All" that might give you what you want here. Switch between these modes by clicking on the boxes that have the purple cylinders running through them.
It also sounds like you have a part that does not have any surfaces square to where you want to begin your cut. You can place additional planes anywhere you want to make up for this. There is a Planes group directly to the right of the Select button on the Home tab in the ribbon. Click the dropdown button and you can see many, many plane types you can create.