From b4013659a81a48908eb3e060f04143ba1f9689bb Mon Sep 17 00:00:00 2001 From: Maximilian Hils Date: Sun, 6 Sep 2015 03:20:58 +0200 Subject: docs :tada: --- docs/dev/architecture.rst | 14 +++ docs/dev/sslkeylogfile.rst | 14 +++ docs/dev/testing.rst | 46 ++++++++ docs/features/passthrough.rst | 94 +++++++++++++++ docs/features/proxyauth.rst | 17 +++ docs/features/responsestreaming.rst | 68 +++++++++++ docs/features/reverseproxy.rst | 56 +++++++++ docs/features/setheaders.rst | 3 +- docs/features/socksproxy.rst | 10 ++ docs/features/sticky.rst | 41 +++++++ docs/features/tcpproxy.rst | 30 +++++ docs/features/upstreamcerts.rst | 21 +++- docs/features/upstreamproxy.rst | 12 ++ docs/index.rst | 31 ++++- docs/modes.rst | 2 +- docs/scripting/inlinescripts.rst | 2 +- docs/scripting/libmproxy.rst | 2 +- docs/transparent.rst | 20 +++- docs/transparent/linux.rst | 45 ++++++++ docs/transparent/osx.rst | 70 +++++++++++ docs/tutorials/30second.rst | 66 +++++++++++ docs/tutorials/gamecenter.rst | 128 +++++++++++++++++++++ docs/tutorials/leaderboard.png | Bin 0 -> 448368 bytes docs/tutorials/one.png | Bin 0 -> 141443 bytes docs/tutorials/supermega.png | Bin 0 -> 92889 bytes docs/tutorials/transparent-dhcp.rst | 87 ++++++++++++++ docs/tutorials/transparent-dhcp/step1_proxy.png | Bin 0 -> 246413 bytes .../tutorials/transparent-dhcp/step1_vbox_eth0.png | Bin 0 -> 28237 bytes .../tutorials/transparent-dhcp/step1_vbox_eth1.png | Bin 0 -> 28765 bytes .../transparent-dhcp/step2_proxied_vm.png | Bin 0 -> 36775 bytes 30 files changed, 872 insertions(+), 7 deletions(-) create mode 100644 docs/dev/architecture.rst create mode 100644 docs/dev/sslkeylogfile.rst create mode 100644 docs/dev/testing.rst create mode 100644 docs/features/passthrough.rst create mode 100644 docs/features/proxyauth.rst create mode 100644 docs/features/responsestreaming.rst create mode 100644 docs/features/reverseproxy.rst create mode 100644 docs/features/socksproxy.rst create mode 100644 docs/features/sticky.rst create mode 100644 docs/features/tcpproxy.rst create mode 100644 docs/features/upstreamproxy.rst create mode 100644 docs/transparent/linux.rst create mode 100644 docs/transparent/osx.rst create mode 100644 docs/tutorials/30second.rst create mode 100644 docs/tutorials/gamecenter.rst create mode 100644 docs/tutorials/leaderboard.png create mode 100644 docs/tutorials/one.png create mode 100644 docs/tutorials/supermega.png create mode 100644 docs/tutorials/transparent-dhcp.rst create mode 100644 docs/tutorials/transparent-dhcp/step1_proxy.png create mode 100644 docs/tutorials/transparent-dhcp/step1_vbox_eth0.png create mode 100644 docs/tutorials/transparent-dhcp/step1_vbox_eth1.png create mode 100644 docs/tutorials/transparent-dhcp/step2_proxied_vm.png (limited to 'docs') diff --git a/docs/dev/architecture.rst b/docs/dev/architecture.rst new file mode 100644 index 00000000..ba902fab --- /dev/null +++ b/docs/dev/architecture.rst @@ -0,0 +1,14 @@ +.. _architecture: + +Architecture +============ + +To give you a better understanding of how mitmproxy works, mitmproxy's +high-level architecture is detailed in the following graphic: + +.. image:: ../schematics/architecture.png + +:download:`architecture.pdf <../schematics/architecture.pdf>` + +Please don't refrain from asking any further +questions on the mailing list, the Slack channel or the GitHub issue tracker. \ No newline at end of file diff --git a/docs/dev/sslkeylogfile.rst b/docs/dev/sslkeylogfile.rst new file mode 100644 index 00000000..d61a10c0 --- /dev/null +++ b/docs/dev/sslkeylogfile.rst @@ -0,0 +1,14 @@ +.. _sslkeylogfile: + +TLS Master Secrets +================== + +The SSL master keys can be logged by mitmproxy so that external programs can decrypt TLS connections +both from and to the proxy. Key logging is enabled by setting the environment variable +:envvar:`SSLKEYLOGFILE` so that it points to a writable text file. +Recent versions of WireShark can use these log files to decrypt packets. +You can specify the key file path in WireShark via +:samp:`Edit -> Preferences -> Protocols -> SSL -> (Pre)-Master-Secret log filename`. + +Note that :envvar:`SSLKEYLOGFILE` is respected by other programs as well, e.g. Firefox and Chrome. +If this creates any issues, you can set :envvar:`MITMPROXY_SSLKEYLOGFILE` alternatively. \ No newline at end of file diff --git a/docs/dev/testing.rst b/docs/dev/testing.rst new file mode 100644 index 00000000..242e0584 --- /dev/null +++ b/docs/dev/testing.rst @@ -0,0 +1,46 @@ +.. _testing: + +Testing +======= + +All the mitmproxy projects strive to maintain 100% code coverage. In general, +patches and pull requests will be declined unless they're accompanied by a +suitable extension to the test suite. + +Our tests are written for the nose_ test framework. +At the point where you send your pull request, a command like this: + +>>> nosetests --with-cov --cov-report term-missing ./test + +Should give output something like this: + +.. code-block:: none + + > ---------- coverage: platform darwin, python 2.7.2-final-0 -- + > Name Stmts Miss Cover Missing + > ---------------------------------------------------- + > libmproxy/__init__ 0 0 100% + > libmproxy/app 4 0 100% + > libmproxy/cmdline 100 0 100% + > libmproxy/controller 69 0 100% + > libmproxy/dump 150 0 100% + > libmproxy/encoding 39 0 100% + > libmproxy/filt 201 0 100% + > libmproxy/flow 891 0 100% + > libmproxy/proxy 427 0 100% + > libmproxy/script 27 0 100% + > libmproxy/utils 133 0 100% + > libmproxy/version 4 0 100% + > ---------------------------------------------------- + > TOTAL 2045 0 100% + > ---------------------------------------------------- + > Ran 251 tests in 11.864s + + +There are exceptions to the coverage requirement - for instance, much of the +console interface code can't sensibly be unit tested. These portions are +excluded from coverage analysis either in the **.coveragerc** file, or using +**#pragma no-cover** directives. To keep our coverage analysis relevant, we use +these measures as sparingly as possible. + +.. _nose: https://nose.readthedocs.org/en/latest/ \ No newline at end of file diff --git a/docs/features/passthrough.rst b/docs/features/passthrough.rst new file mode 100644 index 00000000..83374955 --- /dev/null +++ b/docs/features/passthrough.rst @@ -0,0 +1,94 @@ +.. _passthrough: + +Ignore Domains +============== + +There are two main reasons why you may want to exempt some traffic from mitmproxy's interception mechanism: + +- **Certificate pinning:** Some traffic is is protected using `Certificate Pinning`_ and + mitmproxy's interception leads to errors. For example, the Twitter app, Windows Update or + the Apple App Store fail to work if mitmproxy is active. +- **Convenience:** You really don't care about some parts of the traffic and just want them to go away. + +If you want to peek into (SSL-protected) non-HTTP connections, check out the :ref:`tcpproxy` feature. +If you want to ignore traffic from mitmproxy's processing because of large response bodies, +take a look at the :ref:`responsestreaming` feature. + +How it works +------------ + +================== ============================= +command-line :option:`--ignore regex` +mitmproxy shortcut :kbd:`o` then :kbd:`I` +================== ============================= + + +mitmproxy allows you to specify a regex which is matched against a ``host:port`` string +(e.g. "example.com:443") to determine hosts that should be excluded. + +There are two important quirks to consider: + +- **In transparent mode, the ignore pattern is matched against the IP.** While we usually infer the + hostname from the Host header if the :option:`--host` argument is passed to mitmproxy, we do not + have access to this information before the SSL handshake. +- In regular mode, explicit HTTP requests are never ignored. [#explicithttp]_ The ignore pattern is + applied on CONNECT requests, which initiate HTTPS or clear-text WebSocket connections. + +Tutorial +-------- + +If you just want to ignore one specific domain, there's usually a bulletproof method to do so: + +1. Run mitmproxy or mitmdump in verbose mode (:option:`-v`) and observe the ``host:port`` + information in the serverconnect messages. mitmproxy will filter on these. +2. Take the ``host:port`` string, surround it with ^ and $, escape all dots (. becomes \\.) + and use this as your ignore pattern: + +.. code-block:: none + :emphasize-lines: 6,7,9 + + >>> mitmdump -v + 127.0.0.1:50588: clientconnect + 127.0.0.1:50588: request + -> CONNECT example.com:443 HTTP/1.1 + 127.0.0.1:50588: Set new server address: example.com:443 + 127.0.0.1:50588: serverconnect + -> example.com:443 + ^C + >>> mitmproxy --ignore ^example\.com:443$ + + +Here are some other examples for ignore patterns: + +.. code-block:: none + + # Exempt traffic from the iOS App Store (the regex is lax, but usually just works): + --ignore apple.com:443 + # "Correct" version without false-positives: + --ignore '^(.+\.)?apple\.com:443$' + + # Ignore example.com, but not its subdomains: + --ignore '^example.com:' + + # Ignore everything but example.com and mitmproxy.org: + --ignore '^(?!example\.com)(?!mitmproxy\.org)' + + # Transparent mode: + --ignore 17\.178\.96\.59:443 + # IP address range: + --ignore 17\.178\.\d+\.\d+:443 + + +.. seealso:: + + - :ref:`tcpproxy` + - :ref:`responsestreaming` + +.. rubric:: Footnotes + +.. [#explicithttp] This stems from an limitation of explicit HTTP proxying: + A single connection can be re-used for multiple target domains - a + ``GET http://example.com/`` request may be followed by a ``GET http://evil.com/`` request on the + same connection. If we start to ignore the connection after the first request, + we would miss the relevant second one. +.. _Certificate Pinning: https://security.stackexchange.com/questions/29988/what-is-certificate-pinning \ No newline at end of file diff --git a/docs/features/proxyauth.rst b/docs/features/proxyauth.rst new file mode 100644 index 00000000..edc428a7 --- /dev/null +++ b/docs/features/proxyauth.rst @@ -0,0 +1,17 @@ +.. _proxyauth: + +Proxy Authentication +==================== + + +Asks the user for authentication before they are permitted to use the proxy. +Authentication headers are stripped from the flows, so they are not passed to +upstream servers. For now, only HTTP Basic authentication is supported. The +proxy auth options are not compatible with the transparent, socks or reverse proxy +mode. + +================== ============================= +command-line :option:`--nonanonymous`, + :option:`--singleuser USER`, + :option:`--htpasswd PATH` +================== ============================= \ No newline at end of file diff --git a/docs/features/responsestreaming.rst b/docs/features/responsestreaming.rst new file mode 100644 index 00000000..50fd0614 --- /dev/null +++ b/docs/features/responsestreaming.rst @@ -0,0 +1,68 @@ +.. _responsestreaming: + +Response Streaming +================== + +By using mitmproxy's streaming feature, response contents can be passed to the client incrementally +before they have been fully received by the proxy. This is especially useful for large binary files +such as videos, where buffering the whole file slows down the client's browser. + +By default, mitmproxy will read the entire response, perform any indicated +manipulations on it and then send the (possibly modified) response to +the client. In some cases this is undesirable and you may wish to "stream" +the reponse back to the client. When streaming is enabled, the response is +not buffered on the proxy but directly sent back to the client instead. + +On the command-line +------------------- + +Streaming can be enabled on the command line for all response bodies exceeding a certain size. +The SIZE argument understands k/m/g suffixes, e.g. 3m for 3 megabytes. + +================== ============================= +command-line :option:`--stream SIZE` +================== ============================= + +.. warning:: + + When response streaming is enabled, **streamed response contents will not be + recorded or preserved in any way.** + +.. note:: + + When response streaming is enabled, the response body cannot be modified by the usual means. + +Customizing Response Streaming +------------------------------ + +You can also use an :ref:`inlinescripts` to customize exactly +which responses are streamed. + +Responses that should be tagged for streaming by setting their ``.stream`` attribute to ``True``: + +.. literalinclude:: ../../examples/stream.py + :caption: examples/stream.py + :language: python + +Implementation Details +---------------------- + +When response streaming is enabled, portions of the code which would have otherwise performed changes +on the response body will see an empty response body instead (:py:data:`netlib.http.CONTENT_MISSING`). +Any modifications will be ignored. + +Streamed responses are usually sent in chunks of 4096 bytes. If the response is sent with a +``Transfer-Encoding: chunked`` header, the response will be streamed one chunk at a time. + +Modifying streamed data +----------------------- + +If the ``.stream`` attribute is callable, ``.stream`` will wrap the generator that yields all chunks. + +.. literalinclude:: ../../examples/stream_modify.py + :caption: examples/stream_modify.py + :language: python + +.. seealso:: + + - :ref:`passthrough` \ No newline at end of file diff --git a/docs/features/reverseproxy.rst b/docs/features/reverseproxy.rst new file mode 100644 index 00000000..87a598ff --- /dev/null +++ b/docs/features/reverseproxy.rst @@ -0,0 +1,56 @@ +.. _reverseproxy: + +Reverse Proxy +============= + +In reverse proxy mode, mitmproxy accepts standard HTTP requests and forwards +them to the specified upstream server. This is in contrast to :ref:`upstreamproxy`, in which +mitmproxy forwards HTTP proxy requests to an upstream proxy server. + +================== ===================================== +command-line :option:`-R http[s]://hostname[:port]` +================== ===================================== + +Here, **scheme** signifies if the proxy should use TLS to connect to the server. +mitmproxy always accepts both encrypted and unencrypted requests and transforms +them to what the server expects. + +.. code-block:: none + + >>> mitmdump -R https://httpbin.org -p 80 + >>> curl http://localhost/ + # requests will be transparently upgraded to TLS by mitmproxy + + >>> mitmdump -R https://httpbin.org -p 443 + >>> curl https://localhost/ + # mitmproxy will use TLS on both ends. + + +Host Header +----------- + +In reverse proxy mode, mitmproxy does not rewrite the host header. While often useful, this +may lead to issues with public web servers. For example, consider the following scenario: + +.. code-block:: none + :emphasize-lines: 5 + + >>> mitmdump -d -R http://example.com/ + >>> curl http://localhost:8080/ + + >> GET https://example.com/ + Host: localhost:8080 + User-Agent: curl/7.35.0 + [...] + + << 404 Not Found 345B + +Since the Host header doesn't match "example.com", an error is returned. +There are two ways to solve this: + +1. Modify the hosts file of your OS so that "example.com" resolves to your proxy's IP. + Then, access example.com directly. Make sure that your proxy can still resolve the original IP + or specify an IP in mitmproxy. +2. Use mitmproxy's :ref:`setheaders` feature to rewrite the host header: ``--setheader :~q:Host:example.com``. + However, keep in mind that absolute URLs within the returned document or HTTP redirects will + cause the client application to bypass the proxy. diff --git a/docs/features/setheaders.rst b/docs/features/setheaders.rst index 0a6c2296..f118e6f8 100644 --- a/docs/features/setheaders.rst +++ b/docs/features/setheaders.rst @@ -7,7 +7,8 @@ This feature lets you specify a set of headers to be added to requests or responses, based on a filter pattern. You can specify these either on the command-line, or through an interactive editor in mitmproxy. -Example: +Example: Set the **Host** header to "example.com" for all requests. + .. code-block:: none mitmdump -R http://example.com --setheader :~q:Host:example.com diff --git a/docs/features/socksproxy.rst b/docs/features/socksproxy.rst new file mode 100644 index 00000000..fb9117f2 --- /dev/null +++ b/docs/features/socksproxy.rst @@ -0,0 +1,10 @@ +.. _socksproxy: + +SOCKS Mode +========== + +In this mode, mitmproxy acts as a SOCKS5 proxy server. + +================== ================= +command-line :option:`--socks` +================== ================= \ No newline at end of file diff --git a/docs/features/sticky.rst b/docs/features/sticky.rst new file mode 100644 index 00000000..e155fb9b --- /dev/null +++ b/docs/features/sticky.rst @@ -0,0 +1,41 @@ +.. _sticky: + +Sticky cookies and auth +======================= + +Sticky cookies +-------------- + +When the sticky cookie option is set, __mitmproxy__ will add the cookie most +recently set by the server to any cookie-less request. Consider a service that +sets a cookie to track the session after authentication. Using sticky cookies, +you can fire up mitmproxy, and authenticate to a service as you usually would +using a browser. After authentication, you can request authenticated resources +through mitmproxy as if they were unauthenticated, because mitmproxy will +automatically add the session tracking cookie to requests. Among other things, +this lets you script interactions with authenticated resources (using tools +like wget or curl) without having to worry about authentication. + +Sticky cookies are especially powerful when used in conjunction with :ref:`clientreplay` - you can +record the authentication process once, and simply replay it on startup every time you need +to interact with the secured resources. + +================== ====================== +command-line :option:`-t FILTER` +mitmproxy shortcut :kbd:`o` then :kbd:`t` +================== ====================== + + +Sticky auth +----------- + +The sticky auth option is analogous to the sticky cookie option, in that HTTP +**Authorization** headers are simply replayed to the server once they have been +seen. This is enough to allow you to access a server resource using HTTP Basic +authentication through the proxy. Note that :program:`mitmproxy` doesn't (yet) support +replay of HTTP Digest authentication. + +================== ====================== +command-line :option:`-u FILTER` +mitmproxy shortcut :kbd:`o` then :kbd:`A` +================== ====================== \ No newline at end of file diff --git a/docs/features/tcpproxy.rst b/docs/features/tcpproxy.rst new file mode 100644 index 00000000..53df8ed6 --- /dev/null +++ b/docs/features/tcpproxy.rst @@ -0,0 +1,30 @@ +.. _tcpproxy: + +TCP Proxy +========= + +WebSockets or other non-HTTP protocols are not supported by mitmproxy yet. However, you can exempt +hostnames from processing, so that mitmproxy acts as a generic TCP forwarder. +This feature is closely related to the :ref:`passthrough` functionality, +but differs in two important aspects: + +- The raw TCP messages are printed to the event log. +- SSL connections will be intercepted. + +Please note that message interception or modification are not possible yet. +If you are not interested in the raw TCP messages, you should use the ignore domains feature. + +How it works +------------ + +================== ====================== +command-line :option:`--tcp HOST` +mitmproxy shortcut :kbd:`o` then :kbd:`T` +================== ====================== + +For a detailed description how the hostname pattern works, please look at the :ref:`passthrough` feature. + +.. seealso:: + + - :ref:`passthrough` + - :ref:`responsestreaming` diff --git a/docs/features/upstreamcerts.rst b/docs/features/upstreamcerts.rst index a287daef..84cfb84e 100644 --- a/docs/features/upstreamcerts.rst +++ b/docs/features/upstreamcerts.rst @@ -1,4 +1,23 @@ .. _upstreamcerts: Upstream Certificates -===================== \ No newline at end of file +===================== + +When mitmproxy receives a connection destined for an SSL-protected service, it +freezes the connection before reading its request data, and makes a connection +to the upstream server to "sniff" the contents of its SSL certificate. The +information gained - the **Common Name** and **Subject Alternative Names** - is +then used to generate the interception certificate, which is sent to the client +so the connection can continue. + +This rather intricate little dance lets us seamlessly generate correct +certificates even if the client has specifed only an IP address rather than the +hostname. It also means that we don't need to sniff additional data to generate +certs in transparent mode. + +Upstream cert sniffing is on by default, and can optionally be turned off. + +================== ============================= +command-line :option:`--no-upstream-cert` +mitmproxy shortcut :kbd:`o` then :kbd:`U` +================== ============================= \ No newline at end of file diff --git a/docs/features/upstreamproxy.rst b/docs/features/upstreamproxy.rst new file mode 100644 index 00000000..e06833c2 --- /dev/null +++ b/docs/features/upstreamproxy.rst @@ -0,0 +1,12 @@ +.. _upstreamproxy: + +Upstream proxy mode +=================== + +In this mode, mitmproxy accepts proxy requests and unconditionally forwards all +requests to a specified upstream proxy server. This is in contrast to :ref:`reverseproxy`, +in which mitmproxy forwards ordinary HTTP requests to an upstream server. + +================== =================================== +command-line :option:`-U http://hostname[:port]` +================== =================================== diff --git a/docs/index.rst b/docs/index.rst index 92583075..30fb4027 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -27,8 +27,26 @@ features/filters features/replacements features/clientreplay + features/serverreplay + features/setheaders + features/passthrough + features/proxyauth + features/reverseproxy + features/responsestreaming + features/socksproxy + features/sticky + features/tcpproxy + features/upstreamproxy features/upstreamcerts +.. toctree:: + :hidden: + :caption: Transparent Proxying + + transparent + transparent/linux + transparent/osx + .. toctree:: :hidden: :caption: Scripting @@ -39,8 +57,19 @@ .. toctree:: :hidden: - :caption: Development + :caption: Tutorials + + tutorials/30second + tutorials/gamecenter + tutorials/transparent-dhcp + +.. toctree:: + :hidden: + :caption: Hacking + dev/architecture + dev/testing + dev/sslkeylogfile dev/protocols dev/proxy dev/exceptions diff --git a/docs/modes.rst b/docs/modes.rst index 197c0525..1be7c475 100644 --- a/docs/modes.rst +++ b/docs/modes.rst @@ -91,7 +91,7 @@ In this scenario, we would: 2. Configure the client to use the proxy machine's IP as the default gateway. 3. Quick Check: At this point, you should already be able to visit an unencrypted HTTP site over the proxy. -4. Open the magic domain **mitm.it**mitm and install the certificate +4. Open the magic domain **mitm.it** and install the certificate for your device. Setting the custom gateway on clients can be automated by serving the settings diff --git a/docs/scripting/inlinescripts.rst b/docs/scripting/inlinescripts.rst index 5914def8..19e17582 100644 --- a/docs/scripting/inlinescripts.rst +++ b/docs/scripting/inlinescripts.rst @@ -1,4 +1,4 @@ -.. _inline-scripts: +.. _inlinescripts: Inline Scripts ============== diff --git a/docs/scripting/libmproxy.rst b/docs/scripting/libmproxy.rst index e263b89b..92fa5277 100644 --- a/docs/scripting/libmproxy.rst +++ b/docs/scripting/libmproxy.rst @@ -5,7 +5,7 @@ libmproxy .. note:: - We strongly encourage you to use :ref:`inline-scripts` rather than libmproxy. + We strongly encourage you to use :ref:`inlinescripts` rather than libmproxy. - Inline Scripts are equally powerful and provide an easier syntax. - Most examples are written as inline scripts. - Multiple inline scripts can be used together. diff --git a/docs/transparent.rst b/docs/transparent.rst index fbc94e08..05325283 100644 --- a/docs/transparent.rst +++ b/docs/transparent.rst @@ -3,4 +3,22 @@ Transparent Proxying ==================== -TODO \ No newline at end of file +When a transparent proxy is used, traffic is redirected into a proxy at the +network layer, without any client configuration being required. This makes +transparent proxying ideal for those situations where you can't change client +behaviour - proxy-oblivious Android applications being a common example. + +To set up transparent proxying, we need two new components. The first is a +redirection mechanism that transparently reroutes a TCP connection destined for +a server on the Internet to a listening proxy server. This usually takes the +form of a firewall on the same host as the proxy server - iptables_ on Linux +or pf_ on OSX. When the proxy receives a redirected connection, it sees a vanilla +HTTP request, without a host specification. This is where the second new component +comes in - a host module that allows us to query the redirector for the original +destination of the TCP connection. + +At the moment, mitmproxy supports transparent proxying on OSX Lion and above, +and all current flavors of Linux. + +.. _iptables: http://www.netfilter.org/ +.. _pf: https://en.wikipedia.org/wiki/PF_\(firewall\) \ No newline at end of file diff --git a/docs/transparent/linux.rst b/docs/transparent/linux.rst new file mode 100644 index 00000000..ce79128c --- /dev/null +++ b/docs/transparent/linux.rst @@ -0,0 +1,45 @@ +.. _linux: + +Linux +===== + +On Linux, mitmproxy integrates with the iptables redirection mechanism to +achieve transparent mode. + + 1. :ref:`Install the mitmproxy certificate on the test device ` + + 2. Enable IP forwarding: + + >>> sysctl -w net.ipv4.ip_forward=1 + + You may also want to consider enabling this permanently in ``/etc/sysctl.conf``. + + 3. If your target machine is on the same physical network and you configured it to use a custom + gateway, disable ICMP redirects: + + >>> echo 0 | sudo tee /proc/sys/net/ipv4/conf/*/send_redirects + + You may also want to consider enabling this permanently in ``/etc/sysctl.conf`` + as demonstrated `here `_. + + 4. Create an iptables ruleset that redirects the desired traffic to the + mitmproxy port. Details will differ according to your setup, but the + ruleset should look something like this: + + .. code-block:: none + + iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080 + iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -j REDIRECT --to-port 8080 + + 5. Fire up mitmproxy. You probably want a command like this: + + >>> mitmproxy -T --host + + The :option:`-T` flag turns on transparent mode, and the :option:`--host` + argument tells mitmproxy to use the value of the Host header for URL display. + + 6. Finally, configure your test device to use the host on which mitmproxy is + running as the default gateway. + + +For a detailed walkthrough, have a look at the :ref:`transparent-dhcp` tutorial. diff --git a/docs/transparent/osx.rst b/docs/transparent/osx.rst new file mode 100644 index 00000000..70adc56b --- /dev/null +++ b/docs/transparent/osx.rst @@ -0,0 +1,70 @@ +.. _osx: + +OSX +=== + +OSX Lion integrated the pf_ packet filter from the OpenBSD project, +which mitmproxy uses to implement transparent mode on OSX. +Note that this means we don't support transparent mode for earlier versions of OSX. + + 1. :ref:`Install the mitmproxy certificate on the test device ` + + 2. Enable IP forwarding: + + >>> sudo sysctl -w net.inet.ip.forwarding=1 + + 3. Place the following two lines in a file called, say, **pf.conf**: + + .. code-block:: none + + rdr on en2 inet proto tcp to any port 80 -> 127.0.0.1 port 8080 + rdr on en2 inet proto tcp to any port 443 -> 127.0.0.1 port 8080 + + These rules tell pf to redirect all traffic destined for port 80 or 443 + to the local mitmproxy instance running on port 8080. You should + replace ``en2`` with the interface on which your test device will appear. + + 4. Configure pf with the rules: + + >>> sudo pfctl -f pf.conf + + 5. And now enable it: + + >>>sudo pfctl -e + + 6. Configure sudoers to allow mitmproxy to access pfctl. Edit the file + **/etc/sudoers** on your system as root. Add the following line to the end + of the file: + + .. code-block:: none + + ALL ALL=NOPASSWD: /sbin/pfctl -s state + + Note that this allows any user on the system to run the command + ``/sbin/pfctl -s state`` as root without a password. This only allows + inspection of the state table, so should not be an undue security risk. If + you're special feel free to tighten the restriction up to the user running + mitmproxy. + + 7. Fire up mitmproxy. You probably want a command like this: + + >>> mitmproxy -T --host + + The :option:`-T` flag turns on transparent mode, and the :option:`--host` + argument tells mitmproxy to use the value of the Host header for URL display. + + 8. Finally, configure your test device to use the host on which mitmproxy is + running as the default gateway. + +.. note:: + + Note that the **rdr** rules in the pf.conf given above only apply to inbound + traffic. **This means that they will NOT redirect traffic coming from the box + running pf itself.** We can't distinguish between an outbound connection from a + non-mitmproxy app, and an outbound connection from mitmproxy itself - if you + want to intercept your OSX traffic, you should use an external host to run + mitmproxy. None the less, pf is flexible to cater for a range of creative + possibilities, like intercepting traffic emanating from VMs. See the + **pf.conf** man page for more. + +.. _pf: https://en.wikipedia.org/wiki/PF_\(firewall\) \ No newline at end of file diff --git a/docs/tutorials/30second.rst b/docs/tutorials/30second.rst new file mode 100644 index 00000000..75304bda --- /dev/null +++ b/docs/tutorials/30second.rst @@ -0,0 +1,66 @@ +.. _30second: + +Client playback: a 30 second example +==================================== + +My local cafe is serviced by a rickety and unreliable wireless network, +generously sponsored with ratepayers' money by our city council. After +connecting, you are redirected to an SSL-protected page that prompts you for a +username and password. Once you've entered your details, you are free to enjoy +the intermittent dropouts, treacle-like speeds and incorrectly configured +transparent proxy. + +I tend to automate this kind of thing at the first opportunity, on the theory +that time spent now will be more than made up in the long run. In this case, I +might use Firebug_ to ferret out the form post +parameters and target URL, then fire up an editor to write a little script +using Python's urllib_ to simulate a submission. +That's a lot of futzing about. With mitmproxy we can do the job +in literally 30 seconds, without having to worry about any of the details. +Here's how. + +1. Run mitmdump to record our HTTP conversation to a file. +---------------------------------------------------------- + +>>> mitmdump -w wireless-login + +2. Point your browser at the mitmdump instance. +----------------------------------------------- + +I use a tiny Firefox addon called `Toggle Proxy`_ to switch quickly to and from mitmproxy. +I'm assuming you've already :ref:`configured +your browser with mitmproxy's SSL certificate +authority `. + +3. Log in as usual. +------------------- + +And that's it! You now have a serialized version of the login process in the +file wireless-login, and you can replay it at any time like this: + +>>> mitmdump -c wireless-login + +Embellishments +-------------- + +We're really done at this point, but there are a couple of embellishments we +could make if we wanted. I use wicd_ to +automatically join wireless networks I frequent, and it lets me specify a +command to run after connecting. I used the client replay command above and +voila! - totally hands-free wireless network startup. + +We might also want to prune requests that download CSS, JS, images and so +forth. These add only a few moments to the time it takes to replay, but they're +not really needed and I somehow feel compelled to trim them anyway. So, we fire up +the mitmproxy console tool on our serialized conversation, like so: + +>>> mitmproxy -r wireless-login + +We can now go through and manually delete (using the :kbd:`d` keyboard shortcut) +everything we want to trim. When we're done, we use :kbd:`w` to save the +conversation back to the file. + +.. _Firebug: https://getfirebug.com/ +.. _urllib: https://docs.python.org/library/urllib.html +.. _Toggle Proxy: https://addons.mozilla.org/en-us/firefox/addon/toggle-proxy-51740/ +.. _wicd: https://launchpad.net/wicd \ No newline at end of file diff --git a/docs/tutorials/gamecenter.rst b/docs/tutorials/gamecenter.rst new file mode 100644 index 00000000..18745f2d --- /dev/null +++ b/docs/tutorials/gamecenter.rst @@ -0,0 +1,128 @@ +.. _gamecenter: + +Setting highscores on Apple's GameCenter +======================================== + +The setup +--------- + +In this tutorial, I'm going to show you how simple it is to creatively +interfere with Apple Game Center traffic using mitmproxy. To set things up, +:ref:`install the mitmproxy root certificate `. Then +start mitmproxy on your desktop, and confige the iPhone to use it as a proxy. + + +Taking a look at the Game Center traffic +---------------------------------------- + +Lets take a first look at the Game Center traffic. The game I'll use in this +tutorial is `Super Mega Worm`_ - a great little retro-apocalyptic sidescroller for the iPhone: + +.. image:: supermega.png + :align: center + + +After finishing a game (take your time), watch the traffic flowing through +mitmproxy: + +.. image:: one.png + :align: center + +We see a bunch of things we might expect - initialisation, the retrieval of +leaderboards and so forth. Then, right at the end, there's a POST to this +tantalising URL: + +.. code-block:: none + + https://service.gc.apple.com/WebObjects/GKGameStatsService.woa/wa/submitScore + +The contents of the submission are particularly interesting: + +.. code-block:: xml + + + + + scores + + + category + SMW_Adv_USA1 + context + 0 + score-value + 0 + timestamp + 1363515361321 + + + + + + +This is a `property list`_, containing an identifier for the game, +a score (55, in this case), and a timestamp. Looks pretty simple to mess with. + +Modifying and replaying the score submission +-------------------------------------------- + +Lets edit the score submission. First, select it in mitmproxy, then press +:kbd:`enter` to view it. Make sure you're viewing the request, not the response - +you can use :kbd:`tab` to flick between the two. Now press :kbd:`e` for edit. You'll +be prompted for the part of the request you want to change - press :kbd:`r` for +raw body. Your preferred editor (taken from the EDITOR environment variable) will +now fire up. Lets bump the score up to something a bit more ambitious: + +.. code-block:: xml + + + + + scores + + + category + SMW_Adv_USA1 + context + 0 + score-value + 2200272667 + timestamp + 1363515361321 + + + + + + +Save the file and exit your editor. + +The final step is to replay this modified request. Simply press :kbd:`r` for replay. + + +The glorious result and some intrigue +------------------------------------- + +.. image:: leaderboard.png + :align: center + +And that's it - according to the records, I am the greatest Super Mega Worm +player of all time. + +There's a curious addendum to this tale. When I first wrote this tutorial, all +the top competitors' scores were the same: 2,147,483,647 (this is no longer the +case, beacause there are now so many fellow cheaters using this tutorial). If +you think that number seems familiar, you're right: it's 2^31-1, the maximum +value you can fit into a signed 32-bit int. Now let me tell you another +peculiar thing about Super Mega Worm - at the end of every game, it submits +your highest previous score to the Game Center, not your current score. This +means that it stores your highscore somewhere, and I'm guessing that it reads +that stored score back into a signed integer. So, if you _were_ to cheat by the +relatively pedestrian means of modifying the saved score on your jailbroken +phone, then 2^31-1 might well be the maximum score you could get. Then again, +if the game itself stores its score in a signed 32-bit int, you could get the +same score through perfect play, effectively beating the game. So, which is it +in this case? I'll leave that for you to decide. + +.. _Super Mega Worm: https://itunes.apple.com/us/app/super-mega-worm/id388541990?mt=8 +.. _property list: https://en.wikipedia.org/wiki/Property_list \ No newline at end of file diff --git a/docs/tutorials/leaderboard.png b/docs/tutorials/leaderboard.png new file mode 100644 index 00000000..c1be8df5 Binary files /dev/null and b/docs/tutorials/leaderboard.png differ diff --git a/docs/tutorials/one.png b/docs/tutorials/one.png new file mode 100644 index 00000000..78a636cf Binary files /dev/null and b/docs/tutorials/one.png differ diff --git a/docs/tutorials/supermega.png b/docs/tutorials/supermega.png new file mode 100644 index 00000000..d416f71f Binary files /dev/null and b/docs/tutorials/supermega.png differ diff --git a/docs/tutorials/transparent-dhcp.rst b/docs/tutorials/transparent-dhcp.rst new file mode 100644 index 00000000..1e53b333 --- /dev/null +++ b/docs/tutorials/transparent-dhcp.rst @@ -0,0 +1,87 @@ +.. _transparent-dhcp: + +Transparently proxify virtual machines +====================================== + +This walkthrough illustrates how to set up transparent proxying with mitmproxy. +We use VirtualBox VMs with an Ubuntu proxy machine in this example, +but the general *Internet <--> Proxy VM <--> (Virtual) Internal Network* setup can be applied to other setups. + +1. Configure Proxy VM +--------------------- + +On the proxy machine, **eth0** is connected to the internet. **eth1** is connected to the internal +network that will be proxified and configured to use a static ip (192.168.3.1). + +VirtualBox configuration +^^^^^^^^^^^^^^^^^^^^^^^^ + +.. image:: transparent-dhcp/step1_vbox_eth0.png + +.. image:: transparent-dhcp/step1_vbox_eth1.png + +VM Network Configuration +^^^^^^^^^^^^^^^^^^^^^^^^ + +.. image:: transparent-dhcp/step1_proxy.png + :align: center + +2. Configure DHCP and DNS +------------------------- + +We use dnsmasq to provide DHCP and DNS in our internal network. +Dnsmasq is a lightweight server designed to provide DNS (and optionally +DHCP and TFTP) services to a small-scale network. + +- Before we get to that, we need to fix some Ubuntu quirks: + **Ubuntu >12.04** runs an internal dnsmasq instance (listening on loopback only) by default + `[1] `_. For our use case, this needs to be + disabled by changing ``dns=dnsmasq`` to ``#dns=dnsmasq`` in **/etc/NetworkManager/NetworkManager.conf** + and running + + >>> sudo restart network-manager + + afterwards. +- Now, dnsmasq can be be installed and configured: + + >>> sudo apt-get install dnsmasq + + Replace **/etc/dnsmasq.conf** with the following configuration: + + .. code-block:: none + + # Listen for DNS requests on the internal network + interface=eth1 + # Act as a DHCP server, assign IP addresses to clients + dhcp-range=192.168.3.10,192.168.3.100,96h + # Broadcast gateway and dns server information + dhcp-option=option:router,192.168.3.1 + dhcp-option=option:dns-server,192.168.3.1 + + Apply changes: + + >>> sudo service dnsmasq restart + + Your **proxied machine** in the internal virtual network should now receive an IP address via DHCP: + + .. image:: transparent-dhcp/step2_proxied_vm.png + +3. Redirect traffic to mitmproxy +------------------------------------------ + +To redirect traffic to mitmproxy, we need to add two iptables rules: + +.. code-block:: none + + iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 80 -j REDIRECT --to-port 8080 + iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 443 -j REDIRECT --to-port 8080 + +4. Run mitmproxy +---------------- + +Finally, we can run mitmproxy in transparent mode with + +>>> mitmproxy -T + +The proxied machine cannot to leak any data outside of HTTP or DNS requests. +If required, you can now :ref:`install the mitmproxy certificates on the proxied machine `. \ No newline at end of file diff --git a/docs/tutorials/transparent-dhcp/step1_proxy.png b/docs/tutorials/transparent-dhcp/step1_proxy.png new file mode 100644 index 00000000..a0c94484 Binary files /dev/null and b/docs/tutorials/transparent-dhcp/step1_proxy.png differ diff --git a/docs/tutorials/transparent-dhcp/step1_vbox_eth0.png b/docs/tutorials/transparent-dhcp/step1_vbox_eth0.png new file mode 100644 index 00000000..4b7b4e9b Binary files /dev/null and b/docs/tutorials/transparent-dhcp/step1_vbox_eth0.png differ diff --git a/docs/tutorials/transparent-dhcp/step1_vbox_eth1.png b/docs/tutorials/transparent-dhcp/step1_vbox_eth1.png new file mode 100644 index 00000000..b994d4cb Binary files /dev/null and b/docs/tutorials/transparent-dhcp/step1_vbox_eth1.png differ diff --git a/docs/tutorials/transparent-dhcp/step2_proxied_vm.png b/docs/tutorials/transparent-dhcp/step2_proxied_vm.png new file mode 100644 index 00000000..2046cc57 Binary files /dev/null and b/docs/tutorials/transparent-dhcp/step2_proxied_vm.png differ -- cgit v1.2.3