diff options
| -rw-r--r-- | docs/features/passthrough.rst | 4 | ||||
| -rw-r--r-- | docs/features/responsestreaming.rst | 68 | ||||
| -rw-r--r-- | docs/features/streaming.rst | 103 | ||||
| -rw-r--r-- | docs/features/tcpproxy.rst | 2 | ||||
| -rw-r--r-- | docs/index.rst | 2 | ||||
| -rw-r--r-- | examples/complex/stream.py | 2 |
6 files changed, 108 insertions, 73 deletions
diff --git a/docs/features/passthrough.rst b/docs/features/passthrough.rst index d68a49a9..00462e9d 100644 --- a/docs/features/passthrough.rst +++ b/docs/features/passthrough.rst @@ -16,7 +16,7 @@ mechanism: If you want to peek into (SSL-protected) non-HTTP connections, check out the :ref:`tcpproxy` feature. If you want to ignore traffic from mitmproxy's processing because of large response bodies, -take a look at the :ref:`responsestreaming` feature. +take a look at the :ref:`streaming` feature. How it works ------------ @@ -89,7 +89,7 @@ Here are some other examples for ignore patterns: .. seealso:: - :ref:`tcpproxy` - - :ref:`responsestreaming` + - :ref:`streaming` - mitmproxy's "Limit" feature .. rubric:: Footnotes diff --git a/docs/features/responsestreaming.rst b/docs/features/responsestreaming.rst deleted file mode 100644 index 6fa93271..00000000 --- a/docs/features/responsestreaming.rst +++ /dev/null @@ -1,68 +0,0 @@ -.. _responsestreaming: - -Response Streaming -================== - -By using mitmproxy's streaming feature, response contents can be passed to the client incrementally -before they have been fully received by the proxy. This is especially useful for large binary files -such as videos, where buffering the whole file slows down the client's browser. - -By default, mitmproxy will read the entire response, perform any indicated -manipulations on it and then send the (possibly modified) response to -the client. In some cases this is undesirable and you may wish to "stream" -the response back to the client. When streaming is enabled, the response is -not buffered on the proxy but directly sent back to the client instead. - -On the command-line -------------------- - -Streaming can be enabled on the command line for all response bodies exceeding a certain size. -The SIZE argument understands k/m/g suffixes, e.g. 3m for 3 megabytes. - -================== ================= -command-line ``--stream SIZE`` -================== ================= - -.. warning:: - - When response streaming is enabled, **streamed response contents will not be - recorded or preserved in any way.** - -.. note:: - - When response streaming is enabled, the response body cannot be modified by the usual means. - -Customizing Response Streaming ------------------------------- - -You can also use a script to customize exactly which responses are streamed. - -Responses that should be tagged for streaming by setting their ``.stream`` -attribute to ``True``: - -.. literalinclude:: ../../examples/complex/stream.py - :caption: examples/complex/stream.py - :language: python - -Implementation Details ----------------------- - -When response streaming is enabled, portions of the code which would have otherwise performed -changes on the response body will see an empty response body. Any modifications will be ignored. - -Streamed responses are usually sent in chunks of 4096 bytes. If the response is sent with a -``Transfer-Encoding: chunked`` header, the response will be streamed one chunk at a time. - -Modifying streamed data ------------------------ - -If the ``.stream`` attribute is callable, ``.stream`` will wrap the generator that yields all -chunks. - -.. literalinclude:: ../../examples/complex/stream_modify.py - :caption: examples/complex/stream_modify.py - :language: python - -.. seealso:: - - - :ref:`passthrough` diff --git a/docs/features/streaming.rst b/docs/features/streaming.rst new file mode 100644 index 00000000..93f39051 --- /dev/null +++ b/docs/features/streaming.rst @@ -0,0 +1,103 @@ +.. _streaming: + +HTTP Streaming +============== + +By default, mitmproxy will read the entire request/response, perform any indicated +manipulations on it and then send the (possibly modified) request or response to +the server or the client respectively. In some cases this is undesirable and you may wish to "stream" +the request/response. When streaming is enabled, the request/response is +not buffered on the proxy but directly sent to the server/client instead. +Though, HTTP headers are still fully buffered before being sent. + +Response Streaming +------------------ + +By using mitmproxy's streaming feature, response contents can be passed to the client incrementally +before they have been fully received by the proxy. This is especially useful for large binary files +such as videos, where buffering the whole file slows down the client's browser. + +Request Streaming +----------------- + +As the name suggests, request streaming feature can be used to stream request body to the server incrementally +before it has been fully received by the proxy. This may be useful for large file uploads. + + +On the command-line +------------------- + +Streaming can be enabled on the command line for all request and response bodies exceeding a certain size. +The SIZE argument understands k/m/g suffixes, e.g. 3m for 3 megabytes. + +================== ================= +command-line ``--set stream_large_bodies=SIZE`` +================== ================= + +.. warning:: + + When streaming is enabled, **streamed request/response contents will not be + recorded or preserved in any way.** + +.. note:: + + When streaming is enabled, the request/response body cannot be modified by the usual means. + +Customizing Streaming +--------------------- + +You can also use a script to customize exactly which requests or responses are streamed. + +Requests/Responses that should be tagged for streaming by setting their ``.stream`` +attribute to ``True``: + +.. literalinclude:: ../../examples/complex/stream.py + :caption: examples/complex/stream.py + :language: python + +Implementation Details +---------------------- + +When response streaming is enabled, portions of the code which would have otherwise performed +changes on the request/response body will see an empty body. Any modifications will be ignored. + +Streamed bodies are usually sent in chunks of 4096 bytes. If the response is sent with a +``Transfer-Encoding: chunked`` header, the response will be streamed one chunk at a time. + +Modifying streamed data +----------------------- + +If the ``.stream`` attribute is callable, ``.stream`` will wrap the generator that yields all +chunks. + +.. literalinclude:: ../../examples/complex/stream_modify.py + :caption: examples/complex/stream_modify.py + :language: python + +WebSocket Streaming +=================== + +The WebSocket streaming feature can be used to send the frames as soon as they arrive. This can be useful for large binary file transfers. + +On the command-line +------------------- + +Streaming can be enabled on the command line for all WebSocket frames + +================== ================= +command-line ``--set stream_websockets=true`` +================== ================= + +.. note:: + + When Web Socket streaming is enabled, the message payload cannot be modified. + +Implementation Details +---------------------- +When WebSocket streaming is enabled, portions of the code which may perform changes to the WebSocket message payloads will not have +any effect on the actual payload sent to the server as the frames are sent as and when the arrive. +Though the message payload will be stored in the WebSocket Flow unlike request/response streaming where the body is not stored. + +.. seealso:: + + - :ref:`passthrough` diff --git a/docs/features/tcpproxy.rst b/docs/features/tcpproxy.rst index 0825c024..cba374e3 100644 --- a/docs/features/tcpproxy.rst +++ b/docs/features/tcpproxy.rst @@ -28,4 +28,4 @@ feature. .. seealso:: - :ref:`passthrough` - - :ref:`responsestreaming` + - :ref:`streaming` diff --git a/docs/index.rst b/docs/index.rst index a4e37e71..7cf593ff 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -33,7 +33,7 @@ features/passthrough features/proxyauth features/reverseproxy - features/responsestreaming + features/streaming features/socksproxy features/sticky features/tcpproxy diff --git a/examples/complex/stream.py b/examples/complex/stream.py index 1993cf7f..ae365ec5 100644 --- a/examples/complex/stream.py +++ b/examples/complex/stream.py @@ -1,6 +1,6 @@ def responseheaders(flow): """ Enables streaming for all responses. - This is equivalent to passing `--stream 0` to mitmproxy. + This is equivalent to passing `--set stream_large_bodies=1` to mitmproxy. """ flow.response.stream = True |