A policy cannot be deleted if it has non-default versions or it is attached to any certificate.
To delete a policy, use the DeletePolicyVersion API to delete all non-default versions of the policy; use the DetachPrincipalPolicy API to detach the policy from any certificate; and then use the DeletePolicy API to delete the policy.
When a policy is deleted using DeletePolicy, its default version is deleted with it.