Even with administrative rights to the folder, users are unable to delete shared folders using Outlook Connector.
When deleting a folder, Outlook attempts to move it from one owner (the person who has shared their folder) to another (the deleting user's Deleted Items folder) and this is not possible. Using a permanent delete should work-around this issue. To permanently delete the shared folder (if you have admin rights to it) select it via left mouse click, then use SHIFT + DEL on your keyboard.