Steps to follow to configure IIS to access a website via IP address.
Add a Binding in IIS
Login into your Window server via Remote Desktop.
Go to Start → Administrative Tools → Internet Information Services (IIS) Manager.
In the Connections pane of IIS, expand the Sites and select the website which you want to access via IP address.
Click on Bindings link and you will see current bindings of that website.
Under the binding tab, Click on Add button to add a new binding.
On the Add Site Binding window, keep website Type as http. Select an IP address from the drop-down menu upon which you want to bind the website. Since other websites (along with their Host Header Values) are already bound on port 80, you won't be able to bind this new website on port 80 without Host Header Value (Host name). So, specify a port number (other than default port 80) on which you want to bind this new website. Keep Host name as blank, click OK and then Close. Once the binding is added in IIS Manager, the next step is allowing a port in Windows Firewall.
Open a Port in Windows Firewall
Go to Start → Administrative Tools → Windows Firewall with Advanced Security.
At Windows Firewall and Advance security window, click on Inbound Rules.
Under Actions pane, click on New Rule and New Inbound Rule Wizard will be opened. On this window, select the Port radio button and click on Next.
On the next screen, select TCP and Specific local ports radio button. Specify a port number (upon which you set binding in IIS) in Specific local ports field and click Next.
On the next screen, select Allow the connection and click Next.
Select the profile where you want to apply this rule and click Next.
Specify Name and Description for the newly created rule & click Finish.
Once you allow the port number in Windows Firewall, you will be able to access your website using via IP address like http://VPS-IP-Address:81.