# Requests #### Source {% embed url="" %} Requests ### The GET Request [HTTP methods](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#Request_methods) such as `GET` and `POST`, determine which action you’re trying to perform when making an HTTP request. ### The Response A `Response` is a powerful object for inspecting the results of the request. ```python response = requests.get('https://api.github.com') ``` #### Status Codes The first bit of information that you can gather from `Response` is the status code. A status code informs you of the status of the request. For example, a `200 OK` status means that your request was successful, whereas a `404 NOT FOUND` status means that the resource you were looking for was not found. There are [many other possible status codes](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes) as well to give you specific insights into what happened with your request. ```python >>> response.status_code 200 ``` `.status_code` returned a `200`, which means your request was successful and the server responded with the data you were requesting. Sometimes, you might want to use this information to make decisions in your code: ```python if response.status_code == 200: print('Success!') elif response.status_code == 404: print('Not Found.') ``` `requests` goes one step further in simplifying this process for you. If you use a `Response` instance in a conditional expression, it will evaluate to `True` if the status code was between `200` and `400`, and `False` otherwise. ```python if response: print('Success!') else: print('An error has occurred.') ``` Keep in mind that this method is *not* verifying that the status code is equal to `200`. The reason for this is that other status codes within the `200` to `400` range, such as `204 NO CONTENT` and `304 NOT MODIFIED`, are also considered successful in the sense that they provide some workable response. For example, the `204` tells you that the response was successful, but there’s no content to return in the message body. So, make sure you use this convenient shorthand only if you want to know if the request was generally successful and then, if necessary, handle the response appropriately based on the status code. Let’s say you don’t want to check the response’s status code in an `if` statement. Instead, you want to raise an exception if the request was unsuccessful. You can do this using `.raise_for_status()`: ```python import requests from requests.exceptions import HTTPError for url in ['https://api.github.com', 'https://api.github.com/invalid']: try: response = requests.get(url) # If the response was successful, no Exception will be raised response.raise_for_status() except HTTPError as http_err: print(f'HTTP error occurred: {http_err}') # Python 3.6 except Exception as err: print(f'Other error occurred: {err}') # Python 3.6 else: print('Success!') ``` #### Content The response of a `GET` request often has some valuable information, known as a payload, in the message body. Using the attributes and methods of `Response`, you can view the payload in a variety of different formats. To see the response’s content in [`bytes`](https://realpython.com/python-strings/), you use `.content`: ```python response = requests.get('https://api.github.com') response.content ... ``` convert them into a [string](https://realpython.com/python-data-types/) using a character encoding such as [UTF-8](https://en.wikipedia.org/wiki/UTF-8). : ```python response = requests.get('https://api.github.com') response.text '{"current_user_url":"https://api.github.com/user","current_user_authorizations_html_url":"https://github.com/settings/connections/applications{/client_ ``` If you take a look at the response, you’ll see that it is actually serialized JSON content. a simpler way to get a dict is to use `.json()`: ```python >>> response.json() {'current_user_url': 'https://api.github.com/user', 'current_user_auth ``` #### Headers The response headers can give you useful information, such as the content type of the response payload and a time limit on how long to cache the response. To view these headers, access `.headers`: ```python >>> response.headers {'Server': 'GitHub.com', 'Date'..... ``` `.headers` returns a dictionary-like object, allowing you to access header values by key. For example, to see the content type of the response payload, you can access `Content-Type`: ```python >>> response.headers['Content-Type'] 'application/json; charset=utf-8'` ``` The HTTP spec defines headers to be case-insensitive, which means we are able to access these headers without worrying about their capitalization: ```python >>> response.headers['content-type'] 'application/json; charset=utf-8' ``` ### Query String Parameters One common way to customize a `GET` request is to pass values through [query string](https://en.wikipedia.org/wiki/Query_string) parameters in the URL. To do this using `get()`, you pass data to `params`. For example, you can use GitHub’s [Search](https://developer.github.com/v3/search/) API to look for the `requests` library: ```python import requests # Search GitHub's repositories for requests response = requests.get( 'https://api.github.com/search/repositories', params={'q': 'requests+language:python'}, ) # Inspect some attributes of the `requests` repository json_response = response.json() repository = json_response['items'][0] print(f'Repository name: {repository["name"]}') # Python 3.6+ print(f'Repository description: {repository["description"]}') # Python 3.6+ Repository name: grequests Repository description: Requests + Gevent = <3 ``` ### Request Headers To customize headers, you pass a dictionary of HTTP headers to `get()` using the `headers` parameter. For example, you can change your previous search request to highlight matching search terms in the results by specifying the `text-match` media type in the `Accept` header: ```python import requests response = requests.get( 'https://api.github.com/search/repositories', params={'q': 'requests+language:python'}, headers={'Accept': 'application/vnd.github.v3.text-match+json'}, ) # View the new `text-matches` array which provides information # about your search term within the results json_response = response.json() repository = json_response['items'][0] print(f'Text matches: {repository["text_matches"]}') ``` The `Accept` header tells the server what content types your application can handle. In this case, since you’re expecting the matching search terms to be highlighted, you’re using the header value `application/vnd.github.v3.text-match+json`, which is a proprietary GitHub `Accept` header where the content is a special JSON format. ### Other HTTP Methods Aside from `GET`, other popular HTTP methods include `POST`, `PUT`, `DELETE`, `HEAD`, `PATCH`, and `OPTIONS`. `requests` provides a method, with a similar signature to `get()`, for each of these HTTP methods: ```python >>> requests.post('https://httpbin.org/post', data={'key':'value'}) >>> requests.put('https://httpbin.org/put', data={'key':'value'}) >>> requests.delete('https://httpbin.org/delete') >>> requests.head('https://httpbin.org/get') >>> requests.patch('https://httpbin.org/patch', data={'key':'value'}) >>> requests.options('https://httpbin.org/get') ``` Each function call makes a request to the `httpbin` service using the corresponding HTTP method. For each method, you can inspect their responses in the same way you did before: ```python >>> response = requests.head('https://httpbin.org/get') >>> response.headers['Content-Type'] 'application/json' >>> response = requests.delete('https://httpbin.org/delete') >>> json_response = response.json() >>> json_response['args'] {} ``` ### The Message Body According to the HTTP specification, `POST`, `PUT`, and the less common `PATCH` requests pass their data through the message body rather than through parameters in the query string. Using `requests`, you’ll pass the payload to the corresponding function’s `data` parameter. `data` takes a dictionary, a list of tuples, bytes, or a file-like object. You’ll want to adapt the data you send in the body of your request to the specific needs of the service you’re interacting with. For example, if your request’s content type is `application/x-www-form-urlencoded`, you can send the form data as a dictionary: ```python >>> requests.post('https://httpbin.org/post', data={'key':'value'}) ``` You can also send that same data as a list of tuples: ```python >>> requests.post('https://httpbin.org/post', data=[('key', 'value')]) ``` If, however, you need to send JSON data, you can use the `json` parameter. When you pass JSON data via `json`, `requests` will serialize your data and add the correct `Content-Type` header for you. [httpbin.org](https://httpbin.org/) is a great resource created by the author of `requests`, [Kenneth Reitz](https://realpython.com/interview-kenneth-reitz/). It’s a service that accepts test requests and responds with data about the requests. For instance, you can use it to inspect a basic `POST` request: ```python >>> response = requests.post('https://httpbin.org/post', json={'key':'value'}) >>> json_response = response.json() >>> json_response['data'] '{"key": "value"}' >>> json_response['headers']['Content-Type'] 'application/json' ``` ### Inspecting Your Request When you make a request, the `requests` library prepares the request before actually sending it to the destination server. Request preparation includes things like validating headers and serializing JSON content. You can view the `PreparedRequest` by accessing `.request`: ```python >>> response = requests.post('https://httpbin.org/post', json={'key':'value'}) >>> response.request.headers['Content-Type'] 'application/json' >>> response.request.url 'https://httpbin.org/post' >>> response.request.body b'{"key": "value"}' ``` ### Authentication Authentication helps a service understand who you are. Typically, you provide your credentials to a server by passing data through the `Authorization` header or a custom header defined by the service. All the request functions you’ve seen to this point provide a parameter called `auth`, which allows you to pass your credentials. One example of an API that requires authentication is GitHub’s [Authenticated User](https://developer.github.com/v3/users/#get-the-authenticated-user) API. This endpoint provides information about the authenticated user’s profile. To make a request to the Authenticated User API, you can pass your GitHub username and password in a tuple to `get()`: ```python from getpass import getpass requests.get('https://api.github.com/', auth=('yzl1994511@163.com', getpass())) ``` When you pass your username and password in a tuple to the `auth` parameter, `requests` is applying the credentials using HTTP’s [Basic access authentication scheme](https://en.wikipedia.org/wiki/Basic_access_authentication) under the hood. Therefore, you could make the same request by passing explicit Basic authentication credentials using `HTTPBasicAuth`: ```python >>> from requests.auth import HTTPBasicAuth >>> from getpass import getpass >>> requests.get( ... 'https://api.github.com/', ... auth=HTTPBasicAuth('yzl1994511@163.com', getpass()) ... ) ``` Though you don’t need to be explicit for Basic authentication, you may want to authenticate using another method. `requests` provides other methods of authentication out of the box such as `HTTPDigestAuth` and `HTTPProxyAuth`. You can even supply your own authentication mechanism. To do so, you must first create a subclass of `AuthBase`. Then, you implement `__call__()`: ```python import requests from requests.auth import AuthBase class TokenAuth(AuthBase): """Implements a custom authentication scheme.""" def __init__(self, token): self.token = token def __call__(self, r): """Attach an API token to a custom auth header.""" r.headers['X-TokenAuth'] = f'{self.token}' # Python 3.6+ return r requests.get('https://httpbin.org/get', auth=TokenAuth('12345abcde-token')) ``` Here, your custom `TokenAuth` mechanism receives a token, then includes that token in the `X-TokenAuth` header of your request. Bad authentication mechanisms can lead to security vulnerabilities, so unless a service requires a custom authentication mechanism for some reason, you’ll always want to use a tried-and-true auth scheme like Basic or OAuth. While you’re thinking about security, let’s consider dealing with SSL Certificates using `requests`. ### SSL Certificate Verification Any time the data you are trying to send or receive is sensitive, security is important. The way that you communicate with secure sites over HTTP is by establishing an encrypted connection using SSL, which means that verifying the target server’s SSL Certificate is critical. The good news is that `requests` does this for you by default. However, there are some cases where you might want to change this behavior. If you want to disable SSL Certificate verification, you pass `False` to the `verify` parameter of the request function: ```python >>> requests.get('https://api.github.com', verify=False) InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning) ``` > **Note:** [`requests` uses a package called `certifi`](http://docs.python-requests.org/en/master/user/advanced/#ca-certificates) to provide Certificate Authorities. This lets `requests` know which authorities it can trust. Therefore, you should update `certifi` frequently to keep your connections as secure as possible. ### Performance When using `requests`, especially in a production application environment, it’s important to consider performance implications. Features like timeout control, sessions, and retry limits can help you keep your application running smoothly. #### Timeouts When you make an inline request to an external service, your system will need to wait upon the response before moving on. If your application waits too long for that response, requests to your service could back up, your user experience could suffer, or your background jobs could hang. By default, `requests` will wait indefinitely on the response, so you should almost always specify a timeout duration to prevent these things from happening. To set the request’s timeout, use the `timeout` parameter. `timeout` can be an integer or float representing the number of seconds to wait on a response before timing out: ```python >>> requests.get('https://api.github.com', timeout=1) >>> requests.get('https://api.github.com', timeout=3.05) ``` In the first request, the request will timeout after 1 second. In the second request, the request will timeout after 3.05 seconds. [You can also pass a tuple](http://docs.python-requests.org/en/master/user/advanced/#timeouts) to `timeout` with the first element being a connect timeout (the time it allows for the client to establish a connection to the server), and the second being a read timeout (the time it will wait on a response once your client has established a connection): ```python >>> requests.get('https://api.github.com', timeout=(2, 5)) ``` If the request establishes a connection within 2 seconds and receives data within 5 seconds of the connection being established, then the response will be returned as it was before. If the request times out, then the function will raise a `Timeout` exception: ```python import requests from requests.exceptions import Timeout try: response = requests.get('https://api.github.com', timeout=1) except Timeout: print('The request timed out') else: print('The request did not time out') ``` #### The Session Object Until now, you’ve been dealing with high level `requests` APIs such as `get()` and `post()`. These functions are abstractions of what’s going on when you make your requests. They hide implementation details such as how connections are managed so that you don’t have to worry about them. Underneath those abstractions is a class called `Session`. If you need to fine-tune your control over how requests are being made or improve the performance of your requests, you may need to use a `Session` instance directly. Sessions are used to persist parameters across requests. For example, if you want to use the same authentication across multiple requests, you could use a session: ```python import requests from getpass import getpass # By using a context manager, you can ensure the resources used by # the session will be released after use with requests.Session() as session: session.auth = ('username', getpass()) # Instead of requests.get(), you'll use session.get() response = session.get('https://api.github.com/user') # You can inspect the response just like you did before print(response.headers) print(response.json()) ``` Each time you make a request with `session`, once it has been initialized with authentication credentials, the credentials will be persisted. The primary performance optimization of sessions comes in the form of persistent connections. When your app makes a connection to a server using a `Session`, it keeps that connection around in a connection pool. When your app wants to connect to the same server again, it will reuse a connection from the pool rather than establishing a new one. #### Max Retries When a request fails, you may want your application to retry the same request. However, `requests` will not do this for you by default. To apply this functionality, you need to implement a custom [Transport Adapter](http://docs.python-requests.org/en/master/user/advanced/#transport-adapters). Transport Adapters let you define a set of configurations per service you’re interacting with. For example, let’s say you want all requests to `https://api.github.com` to retry three times before finally raising a `ConnectionError`. You would build a Transport Adapter, set its `max_retries` parameter, and mount it to an existing `Session`: ```python import requests from requests.adapters import HTTPAdapter from requests.exceptions import ConnectionError github_adapter = HTTPAdapter(max_retries=3) session = requests.Session() # Use `github_adapter` for all requests to endpoints that start with this URL session.mount('https://api.github.com', github_adapter) try: session.get('https://api.github.com') except ConnectionError as ce: print(ce) ``` When you mount the `HTTPAdapter`, `github_adapter`, to `session`, `session` will adhere to its configuration for each request to . Timeouts, Transport Adapters, and sessions are for keeping your code efficient and your application resilient. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://zeliang-yao.gitbook.io/my-note-zeliang-yao/useful/requests.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.