To delete modifiers from a group, click on the circular red 'x' button on the right of every modifier.