To remove contacts, select the contact row and click Remove. You may select more than one contact to remove at a time by pressing Ctrl + Click and then clicking the Remove button.
You may also remove all contacts of a group, by selecting the group and then clicking the Remove Group button.