IP Restrictions and Deployment from Azure Dev Ops

I have IP restrictions on my Integration, Preproduction, and the CMS part of production. These restrictions are managed in the web.config as rewrites. The problem I am having is that I am deploying from Azure Dev Ops using the deployment API. As part of that deployment, it tries to hit the site, which it cant, because of IP restrictions. Azure's IPs are constantly shifting, so how do I manage that.

I found this article:

https://keithjr.dev/development/hosted_build_agent/

At first this failed because 

Add-AzWebAppAccessRestrictionRule

Was not installed. So, I added line to install module Az if it wasnt installed. This failed because some parts of it are already installed. So, how can I have IP restrictions and still use the Deployment API?