Solder mask cutouts can be elliptical or circular. To change to an elliptical/circular cutout click on the Ellipse tab in the cutouts properties in the properties panel.
To edit the parameters of a cutout, first select it. The properties panel will display the properties for the selected cutout.
X
The x coordinate of the center of the cutout
Y
The y coordinate of the center of the cutout
Width
The width of the cutout.
Height
The height of the cutout.
Layer
Select the layer for the cutout