As an update, my solution was to add some Custom CSS to hide the popup when a query ressource (here "whoami"; that must be duplicated accross all pages) does not return "Bad Request" error:
This has to be done at the page level (does not work at the module level, if the module is added to a page).
It seems the Custom CSS is shared accross the pages.
Works great so far in View mode.
However, please note that this prevents the popup to appear in the Editor mode (but one can still authenticate via the query resource "Re-auth" button):