That is odd, it shouldn't be retrying if you have the toggle off and have published the changes.
I have seen in the past that users sometimes forget to publish changes and deploy the changes which leaves them wondering why someone they changed is not occurring
My guess is that either the workflow is being triggered as whole or potentially that a block prior to the specific block in questions is being re-triggered, which is then causing this block to run.
I will need more details about the workflow to troubleshoot further but make sure to triple check that the changes made to turn those toggles off has been made live!