HTTP/1.1 403 Forbidden

We have an app that is communicating with WooCommerce via HTTPS
During development on Windows we were getting “HTTP/1.1 403 Forbidden” and came across this

The solution presented there, setting UserAgent, worked, without exactly explaining in any detail as to why
‘Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20100101 Firefox/12.0’ was a correct and valid value (over the default of ‘Mozilla/3.0 (compatible; Indy Library)’ ).
We now have three cases in our system with “HTTP/1.1 403 Forbidden” and it seems they are all in large commercial environments and that their high level virusscanner/firewall is blocking our request.
This is because one company has conveyed to us that the message they are seeing is
"User Agent “FireFox 12.0(Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20100101 Firefox/12.0)” and
"Not allowed to use this browser”
They are using a product from ZScaler, which is ‘doing’ this, but I am not getting much of a useful response from ZScaler.
Can someone explain, is this because the useragent is somehow identifying an old/buggy browser that should be blocked from working?
If, so what should UserAgent be properly set to, as it seems Windows ‘requires’ a browser be set, yet it seems any setting might eventually cause 403 again when any particular browser setting is discovered to have issues?

I’ve seen this many times before - not just with Indy but with other http client libraries. Sometimes, just changing it from the library default is enough, sometimes you have to mimick an actual browser.

The UserAgent header is meant to be used by servers to do things like handle the differences between platforms and browser capabilities etc - however many firewalls use this to block would be hackers, so if someone attempted hacking in the past using the default Indy Useragent, then that would make it’s way into firewall defences at some point.

So in short, what you did is the correct way to deal with this issue.

You can get your own browser info from WhatIsMyBrowser.com

https://www.whatismybrowser.com/detect/what-is-my-user-agent/

Mine show as:

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/112.0
and
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36

Note Firefox 112.0 … maybe you have a typo ?

Hmm, I get 113.

The 12 is in two places in the stackoverflow link, and it’s ‘code’. Of course that doesn’t mean it’s a repeated boo boo.

Thanks

A Bit more research shows that FireFox 12 was released in 2012

The stackoverflow post was made in 2014 when FireFox 28 was current

FireFox 112 was released 11/4/23

So, I guess you guys are saying I/we/us need to set useragent to some recent browser combination that is unlikely to be blocked by any organization’s firewall, due to bugs, or just plain whimsy, and to periodically update it to be current/recent?

Yet another thing broken on the internet that seems like it will never be fixed.

It would at least be worth trying.
Although web stuff is probably pretty permissive, it would make sense if might get cranky at very old settings.

(Sidenote : My previous Samsung phone is used by my 6yo to watch YouTube Kids … and after running out of battery, it wasn’t working. It was connected to our wifi … but then I noted the system date was wrong by 4 or 5 years, and that was upsetting YouTube until I changed it to the current date/time. )

It might give some context to this discussion if you could see a range of user agents used by WEB Bots on my site over the past 12 months

Status IP Address Hits
200 216.244.66.199 Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com) 93945
403 216.244.66.199 Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com) 42767
200 144.76.72.99 serpstatbot/2.1 (advanced backlink tracking bot; https://serpstatbot.com/; abuse@serpstatbot.com) 7750
403 144.76.72.99 serpstatbot/2.1 (advanced backlink tracking bot; https://serpstatbot.com/; abuse@serpstatbot.com) 6691
403 144.76.68.17 serpstatbot/2.1 (advanced backlink tracking bot; https://serpstatbot.com/; abuse@serpstatbot.com) 6599
403 136.243.155.105 serpstatbot/2.1 (advanced backlink tracking bot; https://serpstatbot.com/; abuse@serpstatbot.com) 6454
403 154.54.249.199 Mozilla/5.0 (compatible; Barkrowler/0.9; +https://babbar.tech/crawler) 5107
200 154.54.249.199 Mozilla/5.0 (compatible; Barkrowler/0.9; +https://babbar.tech/crawler) 5063
200 114.119.129.206 Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot) 3579
200 114.119.140.178 Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot) 3425
200 114.119.140.190 Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot) 3105
504 144.76.72.99 serpstatbot/2.1 (advanced backlink tracking bot; https://serpstatbot.com/; abuse@serpstatbot.com) 3043
200 114.119.140.186 Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot) 2826
403 114.119.140.190 Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot) 2788
403 114.119.129.206 Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot) 2678
403 114.119.140.178 Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot) 2654
200 66.249.79.216 Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) 2598

You might be able to get the full list using the link

http://www.innovasolutions.com.au/cgi-bin/SampleCgi.exe?NhT.yX.b.xAdzId12X.b.xAdzRkWFcnz8I5/m+UNDVET2lva0FjY2Vzc0aSX.b.x?&LnkRefFlg=%26%2347%3BAUSTRALIANMPS%26%2347%3B
but I think it is time limitted.

So go to

Log in “test” “test”

Change Report From To May 2022 Number 12 check “Process Crawler Bots” and hit Change reporting period

Then Hit Web Access Report Served

If you then hit /AUSTRALIANMPS/ you get the full list

Thanks,

The more I look into this the sillier it seems.

Our original validation just used ‘standard rest’ via https, and we never had a single 403 forbidden (with the default useragent)

Now that our website is Wordpress, ‘The system that powers 43% of the internet’ and uses woocommerce, we had to change the useragent just to get out of windows and currently have 4 different corporate clients who are blocking at their firewall.

Could it be that the corporate clients mandate a restricted number of Browsers on their Intranet and block all other browsers trying to exit their network via their firewall?