Solder mask cutouts can be polygonal with optional curved edges. To change to curved cutout click on the Curve tab in the cutouts properties in the properties panel.
To edit the parameters of a cutout, first select it. The properties panel will display the properties for the selected cutout.
Click the 
button to show this help topic.
Vertex
This is the selected vertex/segment. The selected vertex is drawn as a solid red square. The selected segment id drawn in red.
X
The X coordinate of the selected vertex.
Y
The Y coordinate of the selected vertex.
Tension
Slide the 
slider to change the 'curvature' of the curve.
Click 
to add a vertex.
Click 
to delete the selected vertex.
All Segments
Check the Smooth button to smooth the entire curve.
Check the Straight button to straighten the entire curve.
Check the Editable button to make the entire curve editable.
Segment
Click the 
button to straighten the selected segment.
Click the 
button to smooth the selected segment.
Layer
Use this drop-down to set the layer for the rectangle.
Click 
to fillet the polygon
Editable Curves
Drag any of the control points 
to change the shape.
Uncheck the 
button to make the control points for the vertex independent of each other.