Even with administrative rights to the folder, users are unable to delete public folders using Outlook Connector.
When deleting a folder, Outlook attempts to move it from a public space (public folders) to a private one (the user's Deleted Items folder) and this is not possible. Using a permanent delete should work-around this issue. To permanently delete the public folder (if you have admin rights to it) select it via left mouse click, then use SHIFT + DEL on your keyboard.