How To Harden And Simplify Urlopen Function In Python

Python has a useful high-level HTTP client built into the standard library: . While HTTP libraries such as and are excellent, gets things done without external dependencies. urllib.request.urlopen() requests HTTPX urlopen However, there is a notable security concern with : the url can open a variety of protocols, even allowing local filesystem access with URLs. Of course, we should always sanitize user input, but what if we could configure to only load, say, URLs? urlopen file:/// urlopen https The built-in class offers just such an opportunity to streamline , make it more secure, and provide custom error handling. OpenerDirector urlopen Security problems with the default urlopen You might try the following on a Linux system: urllib.request urlopen url = urlopen(url) response: print(response.read().decode()) from import "file:///etc/passwd" with as A little disturbing, yes? . (Perhaps you want to consider scanning with or its related .) Bandit agrees Bandit flake8 plugin The code above certainly communicates the lesson "sanitize your user inputs". Of course, if you control that "url" string, or can ensure that it starts with the correct "https://" scheme, then things are looking better. If this url comes from user input, though, it would be good to check for protocol at least, and, better yet, ensure that the domain is as expected. A suggested solution The following code results in a command that only opens "https://" URLs by default: urlopen urllib.request super().__init__() handlers = handlers ( urllib.request.UnknownHandler, urllib.request.HTTPDefaultErrorHandler, urllib.request.HTTPRedirectHandler, urllib.request.HTTPSHandler, urllib.request.HTTPErrorProcessor, ) handler_class handlers: self.add_handler(handler_class()) opener = SafeOpener() urllib.request.install_opener(opener) import : class SafeOpener (urllib.request.OpenerDirector) : def __init__ (self, handlers: typing.Iterable = None) or for in After running the above, using should fail with a if attempting to open http, ftp, file, data, or any other URL that doesn't have "https:" at the beginning. It will still follow redirects automatically, just as does, and raise an exception for any HTTP status code that isn't in the 200s or 300s. urllib.request.urlopen() URLError urlopen By the way, if you prefer not to override the opener in the function, you could remove the line. Then call anywhere you would have previously used . urlopen install_opener(opener) opener.open() urlopen() The above code assumes that all HTTP calls will be encrypted with TLS (aka "SSL", with "https:" at the beginning of the URL). That also means that testing will need to use "https:" URLs as well. Consider using the library to mock-reproduce HTTP calls, or the library to actually set up certs for testing. If you need to use unencrypted "http:" URLs, though, you can simply add to the iterable. vcr trustme urllib.request.HTTPHandler handlers The handlers, defined This is the chain of default handlers normally used by : urlopen : searches system settings for proxies. If you are 100% sure that your tool or library will never be used with a proxy, then this is not necessary. ProxyHandler : raises a if the protocol requested in the URL is not supported by a handler in this chain. Very helpful and recommended. UnknownHandler URLError : handles unencrypted HTTP connections. Only add this if you are sure you need URL support other than HTTPS HTTPHandler : a prefilter of sorts that turns all responses into exceptions, for handling downstream. This is necessary unless you plan on handling statuses, exceptions, and redirects yourself. HTTPDefaultErrorHandler : handles redirects (status codes 301, 302, 303, or 307) and is necessary if automatic following of redirects is desired. HTTPRedirectHandler : handles ftp: URLs. Not necessary for HTTP calls. FTPHandler : handles file: URLs, and poses security risks. Rarely should this be necessary, if ever. FileHandler : The final response handler, raising any non-200 (OK) responses HTTPErrorProcessor : handles URLs. Hard to imagine why this would be necessary in normal use, and could pose potential security risks with user input. DataHandler data: As may be apparent, several of the above are rarely necessary, if ever, for HTTP API work. Instead, I recommend this list as a happy medium between security and usability: ProxyHandler UnknownHandler HTTPHandler HTTPDefaultErrorHandler HTTPRedirectHandler HTTPSHandler HTTPErrorProcessor Of the above, could possibly be removed if you know you don't need it, and even could be removed if you know that only HTTPS URLs will be called. Actually, this is a pretty good combination: the point of HTTPS is to ensure that nothing is intercepting the connection, and that there are no proxies. So a most-secure list is the same as what is in the example code above: ProxyHandler HTTPHandler UnknownHandler HTTPDefaultErrorHandler HTTPRedirectHandler HTTPSHandler HTTPErrorProcessor Five handlers, not the original nine. Flexibility The use cases for custom instances go beyond just security and simplicity. By subclassing then adding custom status handlers with names like , you create your own handlers that then can be appended to the handler list. These can be used for authorization, retry cadence, and other goals. OpenerDirector BaseHandler http_error_401 I hope these suggestions open up possibilities and peace of mind for you. Also published at https://dev.to/bowmanjd/hardening-and-simplifying-python-s-urlopen-4gee