Every push has a settings button. You have to click on that settings button and remove the website from the permission page.

Did this answer your question?