diff options
Diffstat (limited to 'docs/features')
| -rw-r--r-- | docs/features/anticache.rst | 15 | ||||
| -rw-r--r-- | docs/features/clientreplay.rst | 18 | ||||
| -rw-r--r-- | docs/features/filters.rst | 38 | ||||
| -rw-r--r-- | docs/features/passthrough.rst | 102 | ||||
| -rw-r--r-- | docs/features/proxyauth.rst | 17 | ||||
| -rw-r--r-- | docs/features/replacements.rst | 71 | ||||
| -rw-r--r-- | docs/features/reverseproxy.rst | 43 | ||||
| -rw-r--r-- | docs/features/serverreplay.rst | 52 | ||||
| -rw-r--r-- | docs/features/setheaders.rst | 19 | ||||
| -rw-r--r-- | docs/features/socksproxy.rst | 10 | ||||
| -rw-r--r-- | docs/features/sticky.rst | 41 | ||||
| -rw-r--r-- | docs/features/streaming.rst | 102 | ||||
| -rw-r--r-- | docs/features/upstreamcerts.rst | 23 | ||||
| -rw-r--r-- | docs/features/upstreamproxy.rst | 12 |
14 files changed, 0 insertions, 563 deletions
diff --git a/docs/features/anticache.rst b/docs/features/anticache.rst deleted file mode 100644 index a0c3187a..00000000 --- a/docs/features/anticache.rst +++ /dev/null @@ -1,15 +0,0 @@ -.. _anticache: - -Anticache -========= -When the ``--anticache`` option is passed to mitmproxy, it removes headers -(``if-none-match`` and ``if-modified-since``) that might elicit a -``304 not modified`` response from the server. This is useful when you want to make -sure you capture an HTTP exchange in its totality. It's also often used during -:ref:`clientreplay`, when you want to make sure the server responds with complete data. - - -================== ====================== -command-line ``--anticache`` -mitmproxy shortcut :kbd:`O` then :kbd:`a` -================== ====================== diff --git a/docs/features/clientreplay.rst b/docs/features/clientreplay.rst deleted file mode 100644 index ebe40b5f..00000000 --- a/docs/features/clientreplay.rst +++ /dev/null @@ -1,18 +0,0 @@ -.. _clientreplay: - -Client-side replay -================== - -Client-side replay does what it says on the tin: you provide a previously saved -HTTP conversation, and mitmproxy replays the client requests one by one. Note -that mitmproxy serializes the requests, waiting for a response from the server -before starting the next request. This might differ from the recorded -conversation, where requests may have been made concurrently. - -You may want to use client-side replay in conjunction with the -:ref:`anticache` option, to make sure the server responds with complete data. - -================== =========== -command-line ``-c path`` -mitmproxy shortcut :kbd:`R` then :kbd:`c` -================== =========== diff --git a/docs/features/filters.rst b/docs/features/filters.rst deleted file mode 100644 index e531f734..00000000 --- a/docs/features/filters.rst +++ /dev/null @@ -1,38 +0,0 @@ -.. _filters: - -Filter expressions -================== - -Many commands in :program:`mitmproxy` and :program:`mitmdump` take a filter expression. -Filter expressions consist of the following operators: - -.. documentedlist:: - :header: "Expression" "Description" - :listobject: mitmproxy.flowfilter.help - -- Regexes are Python-style -- Regexes can be specified as quoted strings -- Header matching (~h, ~hq, ~hs) is against a string of the form "name: value". -- Strings with no operators are matched against the request URL. -- The default binary operator is &. - -Examples --------- - -URL containing "google.com": - -.. code-block:: none - - google\.com - -Requests whose body contains the string "test": - -.. code-block:: none - - ~q ~b test - -Anything but requests with a text/html content type: - -.. code-block:: none - - !(~q & ~t "text/html") diff --git a/docs/features/passthrough.rst b/docs/features/passthrough.rst deleted file mode 100644 index 91fcb9b6..00000000 --- a/docs/features/passthrough.rst +++ /dev/null @@ -1,102 +0,0 @@ -.. _passthrough: - -Ignore Domains -============== - -There are two main reasons why you may want to exempt some traffic from mitmproxy's interception -mechanism: - -- **Certificate pinning:** Some traffic is is protected using `Certificate Pinning`_ and - mitmproxy's interception leads to errors. For example, the Twitter app, Windows Update or - the Apple App Store fail to work if mitmproxy is active. -- **Convenience:** You really don't care about some parts of the traffic and just want them to go - away. Note that mitmproxy's "Limit" option is often the better alternative here, as it is - not affected by the limitations listed below. - -If you want to peek into (SSL-protected) non-HTTP connections, check out the :ref:`tcp_proxy` -feature. -If you want to ignore traffic from mitmproxy's processing because of large response bodies, -take a look at the :ref:`streaming` feature. - -How it works ------------- - -================== ====================== -command-line ``--ignore regex`` -mitmproxy shortcut :kbd:`O` then :kbd:`I` -================== ====================== - - -mitmproxy allows you to specify a regex which is matched against a ``host:port`` string -(e.g. "example.com:443") to determine hosts that should be excluded. - -Limitations ------------ - -There are two important quirks to consider: - -- **In transparent mode, the ignore pattern is matched against the IP and ClientHello SNI host.** While we usually infer the - hostname from the Host header if the ``--host`` argument is passed to mitmproxy, we do not - have access to this information before the SSL handshake. If the client uses SNI however, then we treat the SNI host as an ignore target. -- **In regular and upstream proxy mode, explicit HTTP requests are never ignored.** [#explicithttp]_ The ignore pattern is - applied on CONNECT requests, which initiate HTTPS or clear-text WebSocket connections. - -Tutorial --------- - -If you just want to ignore one specific domain, there's usually a bulletproof method to do so: - -1. Run mitmproxy or mitmdump in verbose mode (``-v``) and observe the ``host:port`` - information in the serverconnect messages. mitmproxy will filter on these. -2. Take the ``host:port`` string, surround it with ^ and $, escape all dots (. becomes \\.) - and use this as your ignore pattern: - -.. code-block:: none - :emphasize-lines: 6,7,9 - - >>> mitmdump -v - 127.0.0.1:50588: clientconnect - 127.0.0.1:50588: request - -> CONNECT example.com:443 HTTP/1.1 - 127.0.0.1:50588: Set new server address: example.com:443 - 127.0.0.1:50588: serverconnect - -> example.com:443 - ^C - >>> mitmproxy --ignore ^example\.com:443$ - - -Here are some other examples for ignore patterns: - -.. code-block:: none - - # Exempt traffic from the iOS App Store (the regex is lax, but usually just works): - --ignore apple.com:443 - # "Correct" version without false-positives: - --ignore '^(.+\.)?apple\.com:443$' - - # Ignore example.com, but not its subdomains: - --ignore '^example.com:' - - # Ignore everything but example.com and mitmproxy.org: - --ignore '^(?!example\.com)(?!mitmproxy\.org)' - - # Transparent mode: - --ignore 17\.178\.96\.59:443 - # IP address range: - --ignore 17\.178\.\d+\.\d+:443 - - -.. seealso:: - - - :ref:`tcp_proxy` - - :ref:`streaming` - - mitmproxy's "Limit" feature - -.. rubric:: Footnotes - -.. [#explicithttp] This stems from an limitation of explicit HTTP proxying: - A single connection can be re-used for multiple target domains - a - ``GET http://example.com/`` request may be followed by a ``GET http://evil.com/`` request on the - same connection. If we start to ignore the connection after the first request, - we would miss the relevant second one. -.. _Certificate Pinning: https://security.stackexchange.com/questions/29988/what-is-certificate-pinning diff --git a/docs/features/proxyauth.rst b/docs/features/proxyauth.rst deleted file mode 100644 index afdbb639..00000000 --- a/docs/features/proxyauth.rst +++ /dev/null @@ -1,17 +0,0 @@ -.. _proxyauth: - -Proxy Authentication -==================== - - -Asks the user for authentication before they are permitted to use the proxy. -Authentication headers are stripped from the flows, so they are not passed to -upstream servers. For now, only HTTP Basic authentication is supported. The -proxy auth options are not compatible with the transparent, socks or reverse proxy -mode. - -================== ====================== -command-line ``--nonanonymous``, - ``--singleuser USER``, - ``--htpasswd PATH`` -================== ====================== diff --git a/docs/features/replacements.rst b/docs/features/replacements.rst deleted file mode 100644 index 39dccca2..00000000 --- a/docs/features/replacements.rst +++ /dev/null @@ -1,71 +0,0 @@ -.. _replacements: - -Replacements -============ - -Mitmproxy lets you specify an arbitrary number of patterns that define text -replacements within flows. Each pattern has 3 components: a filter that defines -which flows a replacement applies to, a regular expression that defines what -gets replaced, and a target value that defines what is substituted in. - -Replace hooks fire when either a client request or a server response is -received. Only the matching flow component is affected: so, for example, if a -replace hook is triggered on server response, the replacement is only run on -the Response object leaving the Request intact. You control whether the hook -triggers on the request, response or both using the filter pattern. If you need -finer-grained control than this, it's simple to create a script using the -replacement API on Flow components. - -Replacement hooks are extremely handy in interactive testing of applications. -For instance you can use a replace hook to replace the text "XSS" with a -complicated XSS exploit, and then "inject" the exploit simply by interacting -with the application through the browser. When used with tools like Firebug and -mitmproxy's own interception abilities, replacement hooks can be an amazingly -flexible and powerful feature. - - -On the command-line -------------------- - -The replacement hook command-line options use a compact syntax to make it easy -to specify all three components at once. The general form is as follows: - -.. code-block:: none - - /patt/regex/replacement - -Here, **patt** is a mitmproxy filter expression, **regex** is a valid Python -regular expression, and **replacement** is a string literal. The first -character in the expression (``/`` in this case) defines what the separation -character is. Here's an example of a valid expression that replaces "foo" with -"bar" in all requests: - -.. code-block:: none - - :~q:foo:bar - -In practice, it's pretty common for the replacement literal to be long and -complex. For instance, it might be an XSS exploit that weighs in at hundreds or -thousands of characters. To cope with this, there's a variation of the -replacement hook specifier that lets you load the replacement text from a file. -To specify a file as replacement, prefix the file path with ``@``. -You might start **mitmdump** as follows: - ->>> mitmdump --replacements :~q:foo:@~/xss-exploit - -This will load the replacement text from the file ``~/xss-exploit``. - -The ``--replacements`` flag can be passed multiple times. - - -Interactively -------------- - -The :kbd:`R` shortcut key in the mitmproxy options menu (:kbd:`O`) lets you add and edit -replacement hooks using a built-in editor. The context-sensitive help (:kbd:`?`) has -complete usage information. - -================== ======================= -command-line ``--replacements`` -mitmproxy shortcut :kbd:`O` then :kbd:`R` -================== ======================= diff --git a/docs/features/reverseproxy.rst b/docs/features/reverseproxy.rst deleted file mode 100644 index 57b353ae..00000000 --- a/docs/features/reverseproxy.rst +++ /dev/null @@ -1,43 +0,0 @@ -.. _reverseproxy: - -Reverse Proxy -============= - -In reverse proxy mode, mitmproxy accepts standard HTTP(S) requests and forwards -them to the specified upstream server. This is in contrast to :ref:`upstreamproxy`, in which -mitmproxy forwards HTTP(S) proxy requests to an upstream proxy server. - -================== ================================ -command-line ``-R http[s]://hostname[:port]`` -================== ================================ - -Here, **http[s]** signifies if the proxy should use TLS to connect to the server. -mitmproxy always accepts both encrypted and unencrypted requests and transforms -them to what the server expects. - -.. code-block:: none - - >>> mitmdump -R https://httpbin.org -p 80 - >>> curl http://localhost/ - # requests will be transparently upgraded to TLS by mitmproxy - - >>> mitmdump -R https://httpbin.org -p 443 - >>> curl https://localhost/ - # mitmproxy will use TLS on both ends. - - -Host Header ------------ - -In reverse proxy mode, mitmproxy automatically rewrites the Host header to match the -upstream server. This allows mitmproxy to easily connect to existing endpoints on the -open web (e.g. ``mitmproxy -R https://example.com``). You can disable this behaviour -by passing ``--keep-host-header`` on the console. - -However, keep in mind that absolute URLs within the returned document or HTTP redirects will -NOT be rewritten by mitmproxy. This means that if you click on a link for "http://example.com" -in the returned web page, you will be taken directly to that URL, bypassing mitmproxy. - -One possible way to address this is to modify the hosts file of your OS so that "example.com" -resolves to your proxy's IP, and then access the proxy by going directly to example.com. -Make sure that your proxy can still resolve the original IP, or specify an IP in mitmproxy. diff --git a/docs/features/serverreplay.rst b/docs/features/serverreplay.rst deleted file mode 100644 index aef0296e..00000000 --- a/docs/features/serverreplay.rst +++ /dev/null @@ -1,52 +0,0 @@ -.. _serverreplay: - -Server-side replay -================== - -Server-side replay lets us replay server responses from a saved HTTP -conversation. - -Matching requests with responses --------------------------------- - -By default, :program:`mitmproxy` excludes request headers when matching incoming -requests with responses from the replay file. This works in most circumstances, -and makes it possible to replay server responses in situations where request -headers would naturally vary, e.g. using a different user agent. -The ``--rheader headername`` command-line option allows you to override -this behaviour by specifying individual headers that should be included in matching. - - -Response refreshing -------------------- - -Simply replaying server responses without modification will often result in -unexpected behaviour. For example cookie timeouts that were in the future at -the time a conversation was recorded might be in the past at the time it is -replayed. By default, :program:`mitmproxy` refreshes server responses before sending -them to the client. The **date**, **expires** and **last-modified** headers are -all updated to have the same relative time offset as they had at the time of -recording. So, if they were in the past at the time of recording, they will be -in the past at the time of replay, and vice versa. Cookie expiry times are -updated in a similar way. - -You can turn off response refreshing using the ``--norefresh`` argument, or using -the :kbd:`O` options shortcut within :program:`mitmproxy`. - - -Replaying a session recorded in Reverse-proxy Mode --------------------------------------------------- - -If you have captured the session in reverse proxy mode, in order to replay it you -still have to specify the server URL, otherwise you may get the error: -'HTTP protocol error in client request: Invalid HTTP request form (expected authority or absolute...)'. - -During replay, when the client's requests match previously recorded requests, then the -respective recorded responses are simply replayed by mitmproxy. -Otherwise, the unmatched requests is forwarded to the upstream server. -If forwarding is not desired, you can use the --kill (-k) switch to prevent that. - -================== =========== -command-line ``-S path`` -mitmproxy shortcut :kbd:`R` then :kbd:`s` -================== =========== diff --git a/docs/features/setheaders.rst b/docs/features/setheaders.rst deleted file mode 100644 index 486f8c76..00000000 --- a/docs/features/setheaders.rst +++ /dev/null @@ -1,19 +0,0 @@ -.. _setheaders: - -Set Headers -=========== - -This feature lets you specify a set of headers to be added to requests or -responses, based on a filter pattern. You can specify these either on the -command-line, or through an interactive editor in mitmproxy. - -Example: Set the **Host** header to "example.com" for all requests. - -.. code-block:: none - - mitmdump -R http://example.com --setheader :~q:Host:example.com - -================== ======================= -command-line ``--setheader PATTERN`` -mitmproxy shortcut :kbd:`O` then :kbd:`H` -================== ======================= diff --git a/docs/features/socksproxy.rst b/docs/features/socksproxy.rst deleted file mode 100644 index e1686f45..00000000 --- a/docs/features/socksproxy.rst +++ /dev/null @@ -1,10 +0,0 @@ -.. _socksproxy: - -SOCKS Mode -========== - -In this mode, mitmproxy acts as a SOCKS5 proxy server. - -================== =========== -command-line ``--socks`` -================== =========== diff --git a/docs/features/sticky.rst b/docs/features/sticky.rst deleted file mode 100644 index 5cf32299..00000000 --- a/docs/features/sticky.rst +++ /dev/null @@ -1,41 +0,0 @@ -.. _sticky: - -Sticky cookies and auth -======================= - -Sticky cookies --------------- - -When the sticky cookie option is set, __mitmproxy__ will add the cookie most -recently set by the server to any cookie-less request. Consider a service that -sets a cookie to track the session after authentication. Using sticky cookies, -you can fire up mitmproxy, and authenticate to a service as you usually would -using a browser. After authentication, you can request authenticated resources -through mitmproxy as if they were unauthenticated, because mitmproxy will -automatically add the session tracking cookie to requests. Among other things, -this lets you script interactions with authenticated resources (using tools -like wget or curl) without having to worry about authentication. - -Sticky cookies are especially powerful when used in conjunction with :ref:`clientreplay` - you can -record the authentication process once, and simply replay it on startup every time you need -to interact with the secured resources. - -================== ====================== -command-line ``-t FILTER`` -mitmproxy shortcut :kbd:`O` then :kbd:`t` -================== ====================== - - -Sticky auth ------------ - -The sticky auth option is analogous to the sticky cookie option, in that HTTP -**Authorization** headers are simply replayed to the server once they have been -seen. This is enough to allow you to access a server resource using HTTP Basic -authentication through the proxy. Note that :program:`mitmproxy` doesn't (yet) support -replay of HTTP Digest authentication. - -================== ====================== -command-line ``-u FILTER`` -mitmproxy shortcut :kbd:`O` then :kbd:`A` -================== ====================== diff --git a/docs/features/streaming.rst b/docs/features/streaming.rst deleted file mode 100644 index 82510843..00000000 --- a/docs/features/streaming.rst +++ /dev/null @@ -1,102 +0,0 @@ -.. _streaming: - -HTTP Streaming -============== - -By default, mitmproxy will read the entire request/response, perform any indicated -manipulations on it and then send the (possibly modified) message to -the other party. In some cases this is undesirable and you may wish to "stream" -the request/response. When streaming is enabled, the request/response is -not buffered on the proxy but directly sent to the server/client instead. -HTTP headers are still fully buffered before being sent. - -Request Streaming ------------------ - -Request streaming can be used to incrementally stream a request body to the server -before it has been fully received by the proxy. This is useful for large file uploads. - -Response Streaming ------------------- - -By using mitmproxy's streaming feature, response contents can be passed to the client incrementally -before they have been fully received by the proxy. This is especially useful for large binary files -such as videos, where buffering the whole file slows down the client's browser. - -On the command-line -------------------- - -Streaming can be enabled on the command line for all request and response bodies exceeding a certain size. -The SIZE argument understands k/m/g suffixes, e.g. 3m for 3 megabytes. - -================== ================= -command-line ``--set stream_large_bodies=SIZE`` -================== ================= - -.. warning:: - - When streaming is enabled, **streamed request/response contents will not be - recorded or preserved in any way.** - -.. note:: - - When streaming is enabled, the request/response body cannot be modified by the usual means. - -Customizing Streaming ---------------------- - -You can also use a script to customize exactly which requests or responses are streamed. - -Requests/Responses that should be tagged for streaming by setting their ``.stream`` -attribute to ``True``: - -.. literalinclude:: ../../examples/complex/stream.py - :caption: examples/complex/stream.py - :language: python - -Implementation Details ----------------------- - -When response streaming is enabled, portions of the code which would have otherwise performed -changes on the request/response body will see an empty body. Any modifications will be ignored. - -Streamed bodies are usually sent in chunks of 4096 bytes. If the response is sent with a -``Transfer-Encoding: chunked`` header, the response will be streamed one chunk at a time. - -Modifying streamed data ------------------------ - -If the ``.stream`` attribute is callable, ``.stream`` will wrap the generator that yields all -chunks. - -.. literalinclude:: ../../examples/complex/stream_modify.py - :caption: examples/complex/stream_modify.py - :language: python - -WebSocket Streaming -=================== - -The WebSocket streaming feature can be used to send the frames as soon as they arrive. This can be useful for large binary file transfers. - -On the command-line -------------------- - -Streaming can be enabled on the command line for all WebSocket frames - -================== ================= -command-line ``--set stream_websockets=true`` -================== ================= - -.. note:: - - When Web Socket streaming is enabled, the message payload cannot be modified. - -Implementation Details ----------------------- -When WebSocket streaming is enabled, portions of the code which may perform changes to the WebSocket message payloads will not have -any effect on the actual payload sent to the server as the frames are immediately forwarded to the server. -In contrast to HTTP streaming, where the body is not stored, the message payload will still be stored in the WebSocket Flow. - -.. seealso:: - - - :ref:`passthrough` diff --git a/docs/features/upstreamcerts.rst b/docs/features/upstreamcerts.rst deleted file mode 100644 index 4ef79e1b..00000000 --- a/docs/features/upstreamcerts.rst +++ /dev/null @@ -1,23 +0,0 @@ -.. _upstreamcerts: - -Upstream Certificates -===================== - -When mitmproxy receives a connection destined for an SSL-protected service, it -freezes the connection before reading its request data, and makes a connection -to the upstream server to "sniff" the contents of its SSL certificate. The -information gained - the **Common Name** and **Subject Alternative Names** - is -then used to generate the interception certificate, which is sent to the client -so the connection can continue. - -This rather intricate little dance lets us seamlessly generate correct -certificates even if the client has specified only an IP address rather than the -hostname. It also means that we don't need to sniff additional data to generate -certs in transparent mode. - -Upstream cert sniffing is on by default, and can optionally be turned off. - -================== ====================== -command-line ``--no-upstream-cert`` -mitmproxy shortcut :kbd:`O` then :kbd:`U` -================== ====================== diff --git a/docs/features/upstreamproxy.rst b/docs/features/upstreamproxy.rst deleted file mode 100644 index a4ccf57f..00000000 --- a/docs/features/upstreamproxy.rst +++ /dev/null @@ -1,12 +0,0 @@ -.. _upstreamproxy: - -Upstream proxy mode -=================== - -In this mode, mitmproxy accepts proxy requests and unconditionally forwards all -requests to a specified upstream proxy server. This is in contrast to :ref:`reverseproxy`, -in which mitmproxy forwards ordinary HTTP requests to an upstream server. - -================== ============================= -command-line ``-U http://hostname[:port]`` -================== ============================= |