diff options
62 files changed, 828 insertions, 29 deletions
diff --git a/doc-src/dev-docs/index.rst b/doc-src/dev-docs/index.rst deleted file mode 100644 index 15ec3617..00000000 --- a/doc-src/dev-docs/index.rst +++ /dev/null @@ -1,24 +0,0 @@ -Welcome to mitmproxy's dev documentation! -========================================= - - -Contents: - -.. toctree:: - :maxdepth: 2 - - End User Documentation <https://mitmproxy.org/doc/> - - inlinescripts - protocols - proxy - exceptions - models - - -Indices and tables -================== - -* :ref:`genindex` -* :ref:`modindex` - diff --git a/doc-src/dev-docs/.gitignore b/docs/.gitignore index 69fa449d..69fa449d 100644 --- a/doc-src/dev-docs/.gitignore +++ b/docs/.gitignore diff --git a/doc-src/dev-docs/Makefile b/docs/Makefile index 666a3a86..a22bc8a2 100644 --- a/doc-src/dev-docs/Makefile +++ b/docs/Makefile @@ -192,4 +192,4 @@ pseudoxml: @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml." livehtml: - sphinx-autobuild -b html -z '../../libmproxy' -r '___jb_(old|bak)___$$' $(ALLSPHINXOPTS) $(BUILDDIR)/html
\ No newline at end of file + sphinx-autobuild -b html -z '../libmproxy' -r '___jb_(old|bak)___$$' $(ALLSPHINXOPTS) $(BUILDDIR)/html
\ No newline at end of file diff --git a/docs/certinstall-webapp.png b/docs/certinstall-webapp.png Binary files differnew file mode 100644 index 00000000..10e795cd --- /dev/null +++ b/docs/certinstall-webapp.png diff --git a/docs/certinstall.rst b/docs/certinstall.rst new file mode 100644 index 00000000..f0e71223 --- /dev/null +++ b/docs/certinstall.rst @@ -0,0 +1,173 @@ +.. _certinstall: + +About Certificates +================== + +Introduction +------------ + +Mitmproxy can decrypt encrypted traffic on the fly, as long as the client +trusts its built-in certificate authority. Usually this means that the +mitmproxy CA certificates have to be installed on the client device. + +Quick Setup +----------- + +By far the easiest way to install the mitmproxy certificates is to use the +built-in certificate installation app. To do this, just start mitmproxy and +configure your target device with the correct proxy settings. Now start a +browser on the device, and visit the magic domain **mitm.it**. You should see +something like this: + +.. image:: certinstall-webapp.png + +Click on the relevant icon, follow the setup instructions for the platform +you're on and you are good to go. + + +Installing the mitmproxy CA certificate manually +------------------------------------------------ + +Sometimes using the quick install app is not an option - Java or the iOS +Simulator spring to mind - or you just need to do it manually for some other +reason. Below is a list of pointers to manual certificate installation +documentation for some common platforms. + +The mitmproxy CA cert is located in ``~/.mitmproxy`` after it has been generated at the first +start of mitmproxy. + + +iOS +^^^ + +http://kb.mit.edu/confluence/pages/viewpage.action?pageId=152600377 + +iOS Simulator +^^^^^^^^^^^^^ + +See https://github.com/ADVTOOLS/ADVTrustStore#how-to-use-advtruststore + +Java +^^^^ + +See http://docs.oracle.com/cd/E19906-01/820-4916/geygn/index.html + +Android/Android Simulator +^^^^^^^^^^^^^^^^^^^^^^^^^ + +See http://wiki.cacert.org/FAQ/ImportRootCert#Android_Phones_.26_Tablets + +Windows +^^^^^^^ + +See http://windows.microsoft.com/en-ca/windows/import-export-certificates-private-keys#1TC=windows-7 + +Windows (automated) +^^^^^^^^^^^^^^^^^^^ + +>>> certutil.exe -importpfx mitmproxy-ca-cert.p12 + +See also: https://technet.microsoft.com/en-us/library/cc732443.aspx + +Mac OS X +^^^^^^^^ + +See https://support.apple.com/kb/PH7297?locale=en_US + +Ubuntu/Debian +^^^^^^^^^^^^^ + +See http://askubuntu.com/questions/73287/how-do-i-install-a-root-certificate/94861#94861 + +Mozilla Firefox +^^^^^^^^^^^^^^^ + +See https://wiki.mozilla.org/MozillaRootCertificate#Mozilla_Firefox + +Chrome on Linux +^^^^^^^^^^^^^^^ + +See https://code.google.com/p/chromium/wiki/LinuxCertManagement + + +More on mitmproxy certificates +------------------------------ + +The first time **mitmproxy** or **mitmdump** is run, the mitmproxy Certificate +Authority (CA) is created in the config directory (``~/.mitmproxy`` by default). +This CA is used for on-the-fly generation of dummy certificates for each of the +SSL sites that your client visits. Since your browser won't trust the +mitmproxy CA out of the box , you will see an SSL certificate warning every +time you visit a new SSL domain through mitmproxy. When you are testing a +single site through a browser, just accepting the bogus SSL cert manually is +not too much trouble, but there are a many circumstances where you will want to +configure your testing system or browser to trust the mitmproxy CA as a +signing root authority. For security reasons, the mitmproxy CA is generated uniquely on the first +start and is not shared between mitmproxy installations on different devices. + + +CA and cert files +----------------- + +The files created by mitmproxy in the .mitmproxy directory are as follows: + +===================== ==================================================================================== +mitmproxy-ca.pem The certificate **and the private key** in PEM format. +mitmproxy-ca-cert.pem The certificate in PEM format. Use this to distribute on most non-Windows platforms. +mitmproxy-ca-cert.p12 The certificate in PKCS12 format. For use on Windows. +mitmproxy-ca-cert.cer Same file as .pem, but with an extension expected by some Android devices. +===================== ==================================================================================== + +Using a custom certificate +-------------------------- + +You can use your own certificate by passing the ``--cert`` option to +mitmproxy. Mitmproxy then uses the provided certificate for interception of the +specified domains instead of generating a certificate signed by its own CA. + +The certificate file is expected to be in the PEM format. You can include +intermediary certificates right below your leaf certificate, so that you PEM +file roughly looks like this: + +.. code-block:: none + + -----BEGIN PRIVATE KEY----- + <private key> + -----END PRIVATE KEY----- + -----BEGIN CERTIFICATE----- + <cert> + -----END CERTIFICATE----- + -----BEGIN CERTIFICATE----- + <intermediary cert (optional)> + -----END CERTIFICATE----- + + +For example, you can generate a certificate in this format using these instructions: + + +>>> openssl genrsa -out cert.key 2048 +>>> openssl req -new -x509 -key cert.key -out cert.crt + (Specify the mitm domain as Common Name, e.g. *.google.com) +>>> cat cert.key cert.crt > cert.pem +>>> mitmproxy --cert=cert.pem + + +Using a custom certificate authority +------------------------------------ + +By default, mitmproxy will use ``~/.mitmproxy/mitmproxy-ca.pem`` as +the certificate authority to generate certificates for all domains for which no +custom certificate is provided (see above). You can use your own certificate +authority by passing the ``--confdir`` option to mitmproxy. Mitmproxy +will then look for ``mitmproxy-ca.pem`` in the specified directory. If +no such file exists, it will be generated automatically. + + +Using a client side certificate +------------------------------- + +You can use a client certificate by passing the ``--client-certs DIRECTORY`` +option to mitmproxy. If you visit example.org, mitmproxy looks for a file named ``example.org.pem`` +in the specified directory and uses this as the client cert. The certificate file needs to be in +the PEM format and should contain both the unencrypted private key and the certificate. + diff --git a/doc-src/dev-docs/conf.py b/docs/conf.py index 23db112c..65aa19dc 100644 --- a/doc-src/dev-docs/conf.py +++ b/docs/conf.py @@ -149,7 +149,7 @@ html_favicon = "favicon.ico" # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". -html_static_path = ['_static'] +# html_static_path = ['_static'] # Add any extra paths that contain custom files (such as robots.txt or # .htaccess) here, relative to this directory. These files are copied diff --git a/doc-src/dev-docs/exceptions.rst b/docs/dev/exceptions.rst index d1e4bfe5..d1e4bfe5 100644 --- a/doc-src/dev-docs/exceptions.rst +++ b/docs/dev/exceptions.rst diff --git a/doc-src/dev-docs/models.rst b/docs/dev/models.rst index 1ac7d042..1ac7d042 100644 --- a/doc-src/dev-docs/models.rst +++ b/docs/dev/models.rst diff --git a/doc-src/dev-docs/protocols.rst b/docs/dev/protocols.rst index 498f634d..498f634d 100644 --- a/doc-src/dev-docs/protocols.rst +++ b/docs/dev/protocols.rst diff --git a/doc-src/dev-docs/proxy.rst b/docs/dev/proxy.rst index e772e5b9..c0cdb259 100644 --- a/doc-src/dev-docs/proxy.rst +++ b/docs/dev/proxy.rst @@ -1,7 +1,7 @@ -.. _exceptions: +.. _proxy: -Proxy -===== +Proxy Server +============ .. automodule:: libmproxy.proxy diff --git a/doc-src/dev-docs/_static/favicon.ico b/docs/favicon.ico Binary files differindex 3c3b891c..3c3b891c 100644 --- a/doc-src/dev-docs/_static/favicon.ico +++ b/docs/favicon.ico diff --git a/docs/features/upstreamcerts.rst b/docs/features/upstreamcerts.rst new file mode 100644 index 00000000..a287daef --- /dev/null +++ b/docs/features/upstreamcerts.rst @@ -0,0 +1,4 @@ +.. _upstreamcerts: + +Upstream Certificates +=====================
\ No newline at end of file diff --git a/docs/howmitmproxy.rst b/docs/howmitmproxy.rst new file mode 100644 index 00000000..8bc20792 --- /dev/null +++ b/docs/howmitmproxy.rst @@ -0,0 +1,238 @@ +How mitmproxy works +=================== + +Mitmproxy is an enormously flexible tool. Knowing exactly how the proxying +process works will help you deploy it creatively, and take into account its +fundamental assumptions and how to work around them. This document explains +mitmproxy's proxy mechanism in detail, starting with the simplest unencrypted +explicit proxying, and working up to the most complicated interaction - +transparent proxying of SSL-protected traffic [#ssl]_ in the presence of `Server Name Indication`_. + +Explicit HTTP +------------- + +Configuring the client to use mitmproxy as an explicit proxy is the simplest +and most reliable way to intercept traffic. The proxy protocol is codified in the +`HTTP RFC`_, so the behaviour of both +the client and the server is well defined, and usually reliable. In the +simplest possible interaction with mitmproxy, a client connects directly to the +proxy, and makes a request that looks like this: + +.. code-block:: http + + GET http://example.com/index.html HTTP/1.1 + +This is a proxy GET request - an extended form of the vanilla HTTP GET request +that includes a schema and host specification, and it includes all the +information mitmproxy needs to proceed. + +.. image:: schematics/how-mitmproxy-works-explicit.png + :align: center + +1. The client connects to the proxy and makes a request. +2. Mitmproxy connects to the upstream server and simply forwards the request on. + + +Explicit HTTPS +-------------- + +The process for an explicitly proxied HTTPS connection is quite different. The +client connects to the proxy and makes a request that looks like this: + +.. code-block:: http + + CONNECT example.com:443 HTTP/1.1 + +A conventional proxy can neither view nor manipulate an SSL-encrypted data +stream, so a CONNECT request simply asks the proxy to open a pipe between the +client and server. The proxy here is just a facilitator - it blindly forwards +data in both directions without knowing anything about the contents. The +negotiation of the SSL connection happens over this pipe, and the subsequent +flow of requests and responses are completely opaque to the proxy. + +The MITM in mitmproxy +^^^^^^^^^^^^^^^^^^^^^ + +This is where mitmproxy's fundamental trick comes into play. The MITM in its +name stands for Man-In-The-Middle - a reference to the process we use to +intercept and interfere with these theoretically opaque data streams. The basic +idea is to pretend to be the server to the client, and pretend to be the client +to the server, while we sit in the middle decoding traffic from both sides. The +tricky part is that the `Certificate Authority`_ system is +designed to prevent exactly this attack, by allowing a trusted third-party to +cryptographically sign a server's SSL certificates to verify that they are +legit. If this signature doesn't match or is from a non-trusted party, a secure +client will simply drop the connection and refuse to proceed. Despite the many +shortcomings of the CA system as it exists today, this is usually fatal to +attempts to MITM an SSL connection for analysis. Our answer to this conundrum +is to become a trusted Certificate Authority ourselves. Mitmproxy includes a +full CA implementation that generates interception certificates on the fly. To +get the client to trust these certificates, we :ref:`register mitmproxy as a trusted +CA with the device manually <certinstall>`. + +Complication 1: What's the remote hostname? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To proceed with this plan, we need to know the domain name to use in the +interception certificate - the client will verify that the certificate is for +the domain it's connecting to, and abort if this is not the case. At first +blush, it seems that the CONNECT request above gives us all we need - in this +example, both of these values are "example.com". But what if the client had +initiated the connection as follows: + +.. code-block:: http + + CONNECT 10.1.1.1:443 HTTP/1.1 + +Using the IP address is perfectly legitimate because it gives us enough +information to initiate the pipe, even though it doesn't reveal the remote +hostname. + +Mitmproxy has a cunning mechanism that smooths this over - :ref:`upstream +certificate sniffing <upstreamcerts>`. As soon as we +see the CONNECT request, we pause the client part of the conversation, and +initiate a simultaneous connection to the server. We complete the SSL handshake +with the server, and inspect the certificates it used. Now, we use the Common +Name in the upstream SSL certificates to generate the dummy certificate for the +client. Voila, we have the correct hostname to present to the client, even if +it was never specified. + + +Complication 2: Subject Alternative Name +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Enter the next complication. Sometimes, the certificate Common Name is not, in +fact, the hostname that the client is connecting to. This is because of the +optional `Subject Alternative Name`_ field in the SSL certificate +that allows an arbitrary number of alternative domains to be specified. If the +expected domain matches any of these, the client will proceed, even though the +domain doesn't match the certificate Common Name. The answer here is simple: +when we extract the CN from the upstream cert, we also extract the SANs, and +add them to the generated dummy certificate. + + +Complication 3: Server Name Indication +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +One of the big limitations of vanilla SSL is that each certificate requires its +own IP address. This means that you couldn't do virtual hosting where multiple +domains with independent certificates share the same IP address. In a world +with a rapidly shrinking IPv4 address pool this is a problem, and we have a +solution in the form of the `Server Name Indication`_ extension to +the SSL and TLS protocols. This lets the client specify the remote server name +at the start of the SSL handshake, which then lets the server select the right +certificate to complete the process. + +SNI breaks our upstream certificate sniffing process, because when we connect +without using SNI, we get served a default certificate that may have nothing to +do with the certificate expected by the client. The solution is another tricky +complication to the client connection process. After the client connects, we +allow the SSL handshake to continue until just _after_ the SNI value has been +passed to us. Now we can pause the conversation, and initiate an upstream +connection using the correct SNI value, which then serves us the correct +upstream certificate, from which we can extract the expected CN and SANs. + +Putting it all together +^^^^^^^^^^^^^^^^^^^^^^^ + +Lets put all of this together into the complete explicitly proxied HTTPS flow. + +.. image:: schematics/how-mitmproxy-works-explicit-https.png + :align: center + +1. The client makes a connection to mitmproxy, and issues an HTTP CONNECT request. +2. Mitmproxy responds with a ``200 Connection Established``, as if it has set up the CONNECT pipe. +3. The client believes it's talking to the remote server, and initiates the SSL connection. + It uses SNI to indicate the hostname it is connecting to. +4. Mitmproxy connects to the server, and establishes an SSL connection using the SNI hostname + indicated by the client. +5. The server responds with the matching SSL certificate, which contains the CN and SAN values + needed to generate the interception certificate. +6. Mitmproxy generates the interception cert, and continues the + client SSL handshake paused in step 3. +7. The client sends the request over the established SSL connection. +8. Mitmproxy passes the request on to the server over the SSL connection initiated in step 4. + +Transparent HTTP +---------------- + +When a transparent proxy is used, the HTTP/S connection is redirected into a +proxy at the network layer, without any client configuration being required. +This makes transparent proxying ideal for those situations where you can't +change client behaviour - proxy-oblivious Android applications being a common +example. + +To achieve this, we need to introduce two extra components. The first is a +redirection mechanism that transparently reroutes a TCP connection destined for +a server on the Internet to a listening proxy server. This usually takes the +form of a firewall on the same host as the proxy server - `iptables`_ on Linux or +pf_ on OSX. Once the client has initiated the connection, it makes a vanilla HTTP request, +which might look something like this: + +.. code-block:: http + + GET /index.html HTTP/1.1 + +Note that this request differs from the explicit proxy variation, in that it +omits the scheme and hostname. How, then, do we know which upstream host to +forward the request to? The routing mechanism that has performed the +redirection keeps track of the original destination for us. Each routing +mechanism has a different way of exposing this data, so this introduces the +second component required for working transparent proxying: a host module that +knows how to retrieve the original destination address from the router. In +mitmproxy, this takes the form of a built-in set of +modules_ that know how to talk to each platform's redirection mechanism. +Once we have this information, the process is fairly straight-forward. + +.. image:: schematics/how-mitmproxy-works-transparent.png + :align: center + +1. The client makes a connection to the server. +2. The router redirects the connection to mitmproxy, which is typically listening on a local port + of the same host. Mitmproxy then consults the routing mechanism to establish what the original + destination was. +3. Now, we simply read the client's request... +4. ... and forward it upstream. + +Transparent HTTPS +----------------- + +The first step is to determine whether we should treat an incoming connection +as HTTPS. The mechanism for doing this is simple - we use the routing mechanism +to find out what the original destination port is. By default, we treat all +traffic destined for ports 443 and 8443 as SSL. + +From here, the process is a merger of the methods we've described for +transparently proxying HTTP, and explicitly proxying HTTPS. We use the routing +mechanism to establish the upstream server address, and then proceed as for +explicit HTTPS connections to establish the CN and SANs, and cope with SNI. + +.. image:: schematics/how-mitmproxy-works-transparent-https.png + :align: center + +1. The client makes a connection to the server. +2. The router redirects the connection to mitmproxy, which is typically listening on a local port of + the same host. Mitmproxy then consults the routing mechanism to establish what the original + destination was. +3. The client believes it's talking to the remote server, and initiates the SSL connection. It uses + SNI to indicate the hostname it is connecting to. +4. Mitmproxy connects to the server, and establishes an SSL connection using the SNI hostname + indicated by the client. +5. The server responds with the matching SSL certificate, which contains the CN and SAN values + needed to generate the interception certificate. +6. Mitmproxy generates the interception cert, and continues the client SSL handshake paused in + step 3. +7. The client sends the request over the established SSL connection. +8. Mitmproxy passes the request on to the server over the SSL connection initiated in step 4. + +.. rubric:: Footnotes + +.. [#ssl] I use "SSL" to refer to both SSL and TLS in the generic sense, unless otherwise specified. + +.. _Server Name Indication: https://en.wikipedia.org/wiki/Server_Name_Indication +.. _HTTP RFC: https://tools.ietf.org/html/rfc7230 +.. _Certificate Authority: https://en.wikipedia.org/wiki/Certificate_authority +.. _Subject Alternative Name: https://en.wikipedia.org/wiki/SubjectAltName +.. _iptables: http://www.netfilter.org/ +.. _pf: https://en.wikipedia.org/wiki/PF_\(firewall\) +.. _modules: https://github.com/mitmproxy/mitmproxy/tree/master/libmproxy/platform diff --git a/docs/index.rst b/docs/index.rst new file mode 100644 index 00000000..7c792ea4 --- /dev/null +++ b/docs/index.rst @@ -0,0 +1,43 @@ +.. include:: introduction.rst + + +.. toctree:: + :hidden: + :maxdepth: 1 + + introduction + install + certinstall + howmitmproxy + modes + +.. toctree:: + :hidden: + :caption: Tools + + mitmproxy + mitmdump + +.. toctree:: + :hidden: + :caption: Scripting + + scripting/inlinescripts + scripting/libmproxy + + +.. toctree:: + :hidden: + :caption: Development + + dev/protocols + dev/proxy + dev/exceptions + dev/models + +.. Indices and tables + ================== + + * :ref:`genindex` + * :ref:`modindex` + diff --git a/docs/install.rst b/docs/install.rst new file mode 100644 index 00000000..e0a572af --- /dev/null +++ b/docs/install.rst @@ -0,0 +1,100 @@ +.. _install: + +Installation +============ + +.. _install-ubuntu: + +Installation On Ubuntu +---------------------- + +Ubuntu comes with Python but we need to install pip, python-dev and several libraries. +This was tested on a fully patched installation of Ubuntu 14.04. + +>>> sudo apt-get install python-pip python-dev libffi-dev libssl-dev libxml2-dev libxslt1-dev +>>> sudo pip install mitmproxy + +Once installation is complete you can run :ref:`mitmproxy` or :ref:`mitmdump` from a terminal. + +Installation From Source (Ubuntu) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If you would like to install mitmproxy directly from the master branch on GitHub or would like to +get set up to contribute to the project, install the dependencies as you would for a regular +mitmproxy installation (see :ref:`install-ubuntu`). +Then see the Hacking_ section of the README on GitHub. + + + +Installation On Mac OS X +------------------------ + +The easiest way to get up and running on OSX is to download the pre-built binary packages from +`mitmproxy.org`_. + +There are a few bits of customization you might want to do to make mitmproxy comfortable to use on +OSX. The default color scheme is optimized for a dark background terminal, but you can select a +palette for a light terminal background with the ``--palette`` option. +You can use the OSX **open** program to create a simple and effective ``~/.mailcap`` file to view +request and response bodies: + +.. code-block:: none + + application/*; /usr/bin/open -Wn %s + audio/*; /usr/bin/open -Wn %s + image/*; /usr/bin/open -Wn %s + video/*; /usr/bin/open -Wn %s + +Once installation is complete you can run :ref:`mitmproxy` or :ref:`mitmdump` from a terminal. + + +Installation From Source (Mac OS X) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If you would like to install mitmproxy directly from the master branch on GitHub or would like to +get set up to contribute to the project, there are a few OS X specific things to keep in mind. + +- Make sure that XCode is installed from the App Store, and that the command-line tools have been + downloaded (XCode/Preferences/Downloads). +- If you're running a Python interpreter installed with homebrew (or similar), you may have to + install some dependencies by hand. + +Then see the Hacking_ section of the README on GitHub. + +Installation On Windows +----------------------- + +.. note:: + Please note that mitmdump is the only component of mitmproxy that is supported on Windows at + the moment. + + **There is no interactive user interface on Windows.** + + +First, install the latest version of Python 2.7 from the `Python website`_. +If you already have an older version of Python 2.7 installed, make sure to install pip_ +(pip is included in Python 2.7.9+ by default). + +Next, add Python and the Python Scripts directory to your **PATH** variable. +You can do this easily by running the following in powershell: + +>>> [Environment]::SetEnvironmentVariable("Path", "$env:Path;C:\Python27;C:\Python27\Scripts", "User") + +Now, you can install mitmproxy by running + +>>> pip install mitmproxy + +Once the installation is complete, you can run :ref:`mitmdump` from a command prompt. + +Installation From Source (Windows) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If you would like to install mitmproxy directly from the master branch on GitHub or would like to +get set up to contribute to the project, install Python as outlined above, then see the +Hacking_ section of the README on GitHub. + + +.. _Hacking: https://github.com/mitmproxy/mitmproxy/blob/master/README.mkd#hacking +.. _mitmproxy.org: https://mitmproxy.org/ +.. _`Python website`: https://www.python.org/downloads/windows/ +.. _pip: https://pip.pypa.io/en/latest/installing.html
\ No newline at end of file diff --git a/docs/introduction.rst b/docs/introduction.rst new file mode 100644 index 00000000..6ce6add9 --- /dev/null +++ b/docs/introduction.rst @@ -0,0 +1,26 @@ +Introduction +============ + +**mitmproxy** is an interactive, SSL-capable man-in-the-middle proxy for HTTP +with a console interface. + +**mitmdump** is the command-line version of mitmproxy. Think tcpdump for HTTP. + +**libmproxy** is the library that mitmproxy and mitmdump are built on. + +Documentation, tutorials and distribution packages can be found on the +mitmproxy website: `mitmproxy.org <https://mitmproxy.org/>`_ + + +.. rubric:: Features + + +- Intercept HTTP requests and responses and modify them on the fly. +- Save complete HTTP conversations for later replay and analysis. +- Replay the client-side of an HTTP conversations. +- Replay HTTP responses of a previously recorded server. +- Reverse proxy mode to forward traffic to a specified server. +- Transparent proxy mode on OSX and Linux. +- Make scripted changes to HTTP traffic using Python. +- SSL certificates for interception are generated on the fly. +- And much, much more.
\ No newline at end of file diff --git a/docs/mitmdump.rst b/docs/mitmdump.rst new file mode 100644 index 00000000..c56903ec --- /dev/null +++ b/docs/mitmdump.rst @@ -0,0 +1,4 @@ +.. _mitmdump: + +mitmdump +========
\ No newline at end of file diff --git a/doc-src/dev-docs/_static/mitmproxy-long.png b/docs/mitmproxy-long.png Binary files differindex f9397d1e..f9397d1e 100644 --- a/doc-src/dev-docs/_static/mitmproxy-long.png +++ b/docs/mitmproxy-long.png diff --git a/docs/mitmproxy.rst b/docs/mitmproxy.rst new file mode 100644 index 00000000..b39a8f87 --- /dev/null +++ b/docs/mitmproxy.rst @@ -0,0 +1,4 @@ +.. _mitmproxy: + +mitmproxy +=========
\ No newline at end of file diff --git a/docs/modes.rst b/docs/modes.rst new file mode 100644 index 00000000..ccd1f3ee --- /dev/null +++ b/docs/modes.rst @@ -0,0 +1,193 @@ +.. _modes: + +Modes of Operation +================== + +Mitmproxy has four modes of operation that allow you to use mitmproxy in a +variety of scenarios: + +- **Regular** (the default) +- **Transparent** +- **Reverse Proxy** +- **Upstream Proxy** + + +Now, which one should you pick? Use this flow chart: + +.. image:: schematics/proxy-modes-flowchart.png + :align: center + +Regular Proxy +------------- + +Mitmproxy's regular mode is the simplest and the easiest to set up. + +1. Start mitmproxy. +2. Configure your client to use mitmproxy by explicitly setting an HTTP proxy. +3. Quick Check: You should already be able to visit an unencrypted HTTP site through the proxy. +4. Open the magic domain <strong>mitm.it</strong> and install the certificate for your device. + +.. note:: + Unfortunately, some applications bypass the system HTTP proxy settings - Android applications + are a common example. In these cases, you need to use mitmproxy's transparent mode. + +If you are proxying an external device, your network will probably look like this: + +.. image:: schematics/proxy-modes-regular.png + :align: center + +The square brackets signify the source and destination IP addresses. Your +client explicitly connects to mitmproxy and mitmproxy explicitly connects +to the target server. + +Transparent Proxy +----------------- + +In transparent mode, traffic is directed into a proxy at the network layer, +without any client configuration required. This makes transparent proxying +ideal for situations where you can't change client behaviour. In the graphic +below, a machine running mitmproxy has been inserted between the router and +the internet: + +.. image:: schematics/proxy-modes-transparent-1.png + :align: center + +The square brackets signify the source and destination IP addresses. Round +brackets mark the next hop on the *Ethernet/data link* layer. This distinction +is important: when the packet arrives at the mitmproxy machine, it must still +be addressed to the target server. This means that Network Address Translation +should not be applied before the traffic reaches mitmproxy, since this would +remove the target information, leaving mitmproxy unable to determine the real +destination. + +.. image:: schematics/proxy-modes-transparent-wrong.png + :align: center + +Common Configurations +^^^^^^^^^^^^^^^^^^^^^ + +There are many ways to configure your network for transparent proxying. We'll +look at two common scenarios: + +1. Configuring the client to use a custom gateway/router/"next hop" +2. Implementing custom routing on the router + +In most cases, the first option is recommended due to its ease of use. + +(a) Custom Gateway +~~~~~~~~~~~~~~~~~~ + +One simple way to get traffic to the mitmproxy machine with the destination IP +intact, is to simply configure the client with the mitmproxy box as the +default gateway. + +.. image:: schematics/proxy-modes-transparent-2.png + :align: center + +In this scenario, we would: + +1. Configure the proxy machine for transparent mode. You can find instructions + in the :ref:`transparent` section. +2. Configure the client to use the proxy machine's IP as the default gateway. +3. Quick Check: At this point, you should already be able to visit an + unencrypted HTTP site over the proxy. +4. Open the magic domain <strong>mitm.it</strong> and install the certificate + for your device. + +Setting the custom gateway on clients can be automated by serving the settings +out to clients over DHCP. This lets set up an interception network where all +clients are proxied automatically, which can save time and effort. + +.. admonition:: Troubleshooting Transparent Mode + :class: note + + Incorrect transparent mode configurations are a frequent source of + error. If it doesn't work for you, try the following things: + + - Open mitmproxy's event log (press :kbd:`e`) - do you see clientconnect messages? + If not, the packets are not arriving at the proxy. One common cause is the occurrence of ICMP + redirects, which means that your machine is telling the client that there's a faster way to + the internet by contacting your router directly (see the :ref:`transparent` section on how to + disable them). If in doubt, Wireshark_ may help you to see whether something arrives at your + machine or not. + - Make sure you have not explicitly configured an HTTP proxy on the client. + This is not needed in transparent mode. + - Re-check the instructions in the :ref:`transparent` section. Anything you missed? + + If you encounter any other pitfalls that should be listed here, please let us know! + +(b) Custom Routing +~~~~~~~~~~~~~~~~~~ + +In some cases, you may need more fine-grained control of which traffic reaches +the mitmproxy instance, and which doesn't. You may, for instance, choose only +to divert traffic to some hosts into the transparent proxy. There are a huge +number of ways to accomplish this, and much will depend on the router or +packet filter you're using. In most cases, the configuration will look like +this: + +.. image:: schematics/proxy-modes-transparent-3.png + :align: center + + +Reverse Proxy +------------- + +mitmproxy is usually used with a client that uses the proxy to access the +Internet. Using reverse proxy mode, you can use mitmproxy to act like a normal +HTTP server: + +.. image:: schematics/proxy-modes-reverse.png + :align: center + +There are various use-cases: + +- Say you have an internal API running at http://example.local/. You could now + set up mitmproxy in reverse proxy mode at http://debug.example.local/ and + dynamically point clients to this new API endpoint, which provides them + with the same data and you with debug information. Similarly, you could move + your real server to a different IP/port and set up mitmproxy in the original + place to debug and or redirect all sessions. + +- Say you're a web developer working on http://example.com/ (with a development + version running on http://localhost:8000/). You can modify your hosts file so that + example.com points to 127.0.0.1 and then run mitmproxy in reverse proxy mode + on port 80. You can test your app on the example.com domain and get all + requests recorded in mitmproxy. + +- Say you have some toy project that should get SSL support. Simply set up + mitmproxy as a reverse proxy on port 443 and you're done (``mitmdump -p 443 -R + http://localhost:80/``). Mitmproxy auto-detects TLS traffic and intercepts it dynamically. + There are better tools for this specific task, but mitmproxy is very quick and simple way to + set up an SSL-speaking server. + +- Want to add a non-SSL-capable compression proxy in front of your server? You + could even spawn a mitmproxy instance that terminates SSL (``-R http://...``), + point it to the compression proxy and let the compression proxy point to a + SSL-initiating mitmproxy (``-R https://...``), which then points to the real + server. As you see, it's a fairly flexible thing. + +.. admonition:: Caveat: Interactive Use + :class: warning + + Reverse Proxy mode is usually not sufficient to create a copy of an interactive website at + different URL. The HTML served to the client remains unchanged - as soon as the user clicks on + an non-relative URL (or downloads a non-relative image resource), traffic no longer passes + through mitmproxy. + +Upstream Proxy +-------------- + +If you want to chain proxies by adding mitmproxy in front of a different proxy +appliance, you can use mitmproxy's upstream mode. In upstream mode, all +requests are unconditionally transferred to an upstream proxy of your choice. + +.. image:: schematics/proxy-modes-upstream.png + :align: center + +mitmproxy supports both explicit HTTP and explicit HTTPS in upstream proxy +mode. You could in theory chain multiple mitmproxy instances in a row, but +that doesn't make any sense in practice (i.e. outside of our tests). + + +.. _Wireshark: https://wireshark.org/
\ No newline at end of file diff --git a/docs/schematics/architecture.pdf b/docs/schematics/architecture.pdf Binary files differnew file mode 100644 index 00000000..77f5ad58 --- /dev/null +++ b/docs/schematics/architecture.pdf diff --git a/docs/schematics/architecture.png b/docs/schematics/architecture.png Binary files differnew file mode 100644 index 00000000..67d6c718 --- /dev/null +++ b/docs/schematics/architecture.png diff --git a/docs/schematics/architecture.vsdx b/docs/schematics/architecture.vsdx Binary files differnew file mode 100644 index 00000000..c4ff13d2 --- /dev/null +++ b/docs/schematics/architecture.vsdx diff --git a/docs/schematics/how-mitmproxy-works-explicit-https.png b/docs/schematics/how-mitmproxy-works-explicit-https.png Binary files differnew file mode 100644 index 00000000..1f1ca023 --- /dev/null +++ b/docs/schematics/how-mitmproxy-works-explicit-https.png diff --git a/docs/schematics/how-mitmproxy-works-explicit.png b/docs/schematics/how-mitmproxy-works-explicit.png Binary files differnew file mode 100644 index 00000000..c9ba26a7 --- /dev/null +++ b/docs/schematics/how-mitmproxy-works-explicit.png diff --git a/docs/schematics/how-mitmproxy-works-transparent-https.png b/docs/schematics/how-mitmproxy-works-transparent-https.png Binary files differnew file mode 100644 index 00000000..559cddd2 --- /dev/null +++ b/docs/schematics/how-mitmproxy-works-transparent-https.png diff --git a/docs/schematics/how-mitmproxy-works-transparent.png b/docs/schematics/how-mitmproxy-works-transparent.png Binary files differnew file mode 100644 index 00000000..3994d681 --- /dev/null +++ b/docs/schematics/how-mitmproxy-works-transparent.png diff --git a/docs/schematics/proxy-modes-flowchart.png b/docs/schematics/proxy-modes-flowchart.png Binary files differnew file mode 100644 index 00000000..716b5ee2 --- /dev/null +++ b/docs/schematics/proxy-modes-flowchart.png diff --git a/docs/schematics/proxy-modes-regular.png b/docs/schematics/proxy-modes-regular.png Binary files differnew file mode 100644 index 00000000..95bada08 --- /dev/null +++ b/docs/schematics/proxy-modes-regular.png diff --git a/docs/schematics/proxy-modes-reverse.png b/docs/schematics/proxy-modes-reverse.png Binary files differnew file mode 100644 index 00000000..071d3fc8 --- /dev/null +++ b/docs/schematics/proxy-modes-reverse.png diff --git a/docs/schematics/proxy-modes-transparent-1.png b/docs/schematics/proxy-modes-transparent-1.png Binary files differnew file mode 100644 index 00000000..002e0e76 --- /dev/null +++ b/docs/schematics/proxy-modes-transparent-1.png diff --git a/docs/schematics/proxy-modes-transparent-2.png b/docs/schematics/proxy-modes-transparent-2.png Binary files differnew file mode 100644 index 00000000..41997b05 --- /dev/null +++ b/docs/schematics/proxy-modes-transparent-2.png diff --git a/docs/schematics/proxy-modes-transparent-3.png b/docs/schematics/proxy-modes-transparent-3.png Binary files differnew file mode 100644 index 00000000..ee26cb4f --- /dev/null +++ b/docs/schematics/proxy-modes-transparent-3.png diff --git a/docs/schematics/proxy-modes-transparent-wrong.png b/docs/schematics/proxy-modes-transparent-wrong.png Binary files differnew file mode 100644 index 00000000..ca501e93 --- /dev/null +++ b/docs/schematics/proxy-modes-transparent-wrong.png diff --git a/docs/schematics/proxy-modes-upstream.png b/docs/schematics/proxy-modes-upstream.png Binary files differnew file mode 100644 index 00000000..d40a6494 --- /dev/null +++ b/docs/schematics/proxy-modes-upstream.png diff --git a/docs/schematics/proxy-modes.pdf b/docs/schematics/proxy-modes.pdf Binary files differnew file mode 100644 index 00000000..f07ea05e --- /dev/null +++ b/docs/schematics/proxy-modes.pdf diff --git a/docs/schematics/proxy-modes.vsdx b/docs/schematics/proxy-modes.vsdx Binary files differnew file mode 100644 index 00000000..c78cf8d0 --- /dev/null +++ b/docs/schematics/proxy-modes.vsdx diff --git a/docs/screenshots/firefox3-import.jpg b/docs/screenshots/firefox3-import.jpg Binary files differnew file mode 100644 index 00000000..47fcd672 --- /dev/null +++ b/docs/screenshots/firefox3-import.jpg diff --git a/docs/screenshots/firefox3-trust.jpg b/docs/screenshots/firefox3-trust.jpg Binary files differnew file mode 100644 index 00000000..50a2f341 --- /dev/null +++ b/docs/screenshots/firefox3-trust.jpg diff --git a/docs/screenshots/firefox3.jpg b/docs/screenshots/firefox3.jpg Binary files differnew file mode 100644 index 00000000..6c4613b6 --- /dev/null +++ b/docs/screenshots/firefox3.jpg diff --git a/docs/screenshots/ios-gateway.png b/docs/screenshots/ios-gateway.png Binary files differnew file mode 100644 index 00000000..2489cba3 --- /dev/null +++ b/docs/screenshots/ios-gateway.png diff --git a/docs/screenshots/ios-installed.png b/docs/screenshots/ios-installed.png Binary files differnew file mode 100644 index 00000000..2071e441 --- /dev/null +++ b/docs/screenshots/ios-installed.png diff --git a/docs/screenshots/ios-manual.png b/docs/screenshots/ios-manual.png Binary files differnew file mode 100644 index 00000000..3977acfe --- /dev/null +++ b/docs/screenshots/ios-manual.png diff --git a/docs/screenshots/ios-profile.png b/docs/screenshots/ios-profile.png Binary files differnew file mode 100644 index 00000000..5bcd5a0d --- /dev/null +++ b/docs/screenshots/ios-profile.png diff --git a/docs/screenshots/ios-reverse.png b/docs/screenshots/ios-reverse.png Binary files differnew file mode 100644 index 00000000..6ab5b7c0 --- /dev/null +++ b/docs/screenshots/ios-reverse.png diff --git a/docs/screenshots/ios-warning.png b/docs/screenshots/ios-warning.png Binary files differnew file mode 100644 index 00000000..d882c514 --- /dev/null +++ b/docs/screenshots/ios-warning.png diff --git a/docs/screenshots/mitmproxy-flowview.png b/docs/screenshots/mitmproxy-flowview.png Binary files differnew file mode 100644 index 00000000..154963fe --- /dev/null +++ b/docs/screenshots/mitmproxy-flowview.png diff --git a/docs/screenshots/mitmproxy-intercept-filt.png b/docs/screenshots/mitmproxy-intercept-filt.png Binary files differnew file mode 100644 index 00000000..60556ee7 --- /dev/null +++ b/docs/screenshots/mitmproxy-intercept-filt.png diff --git a/docs/screenshots/mitmproxy-intercept-mid.png b/docs/screenshots/mitmproxy-intercept-mid.png Binary files differnew file mode 100644 index 00000000..d5b03922 --- /dev/null +++ b/docs/screenshots/mitmproxy-intercept-mid.png diff --git a/docs/screenshots/mitmproxy-intercept-options.png b/docs/screenshots/mitmproxy-intercept-options.png Binary files differnew file mode 100644 index 00000000..8dc4ad2c --- /dev/null +++ b/docs/screenshots/mitmproxy-intercept-options.png diff --git a/docs/screenshots/mitmproxy-intercept-result.png b/docs/screenshots/mitmproxy-intercept-result.png Binary files differnew file mode 100644 index 00000000..7d9f5c94 --- /dev/null +++ b/docs/screenshots/mitmproxy-intercept-result.png diff --git a/docs/screenshots/mitmproxy-kveditor-editmode.png b/docs/screenshots/mitmproxy-kveditor-editmode.png Binary files differnew file mode 100644 index 00000000..a8315ee5 --- /dev/null +++ b/docs/screenshots/mitmproxy-kveditor-editmode.png diff --git a/docs/screenshots/mitmproxy-kveditor.png b/docs/screenshots/mitmproxy-kveditor.png Binary files differnew file mode 100644 index 00000000..144b9701 --- /dev/null +++ b/docs/screenshots/mitmproxy-kveditor.png diff --git a/docs/screenshots/mitmproxy.png b/docs/screenshots/mitmproxy.png Binary files differnew file mode 100644 index 00000000..42a10e32 --- /dev/null +++ b/docs/screenshots/mitmproxy.png diff --git a/docs/screenshots/osx-addcert-alwaystrust.png b/docs/screenshots/osx-addcert-alwaystrust.png Binary files differnew file mode 100644 index 00000000..4c5cc704 --- /dev/null +++ b/docs/screenshots/osx-addcert-alwaystrust.png diff --git a/docs/screenshots/win7-certstore-trustedroot.png b/docs/screenshots/win7-certstore-trustedroot.png Binary files differnew file mode 100644 index 00000000..e15a87f5 --- /dev/null +++ b/docs/screenshots/win7-certstore-trustedroot.png diff --git a/docs/screenshots/win7-certstore.png b/docs/screenshots/win7-certstore.png Binary files differnew file mode 100644 index 00000000..f8ce54bd --- /dev/null +++ b/docs/screenshots/win7-certstore.png diff --git a/docs/screenshots/win7-wizard.png b/docs/screenshots/win7-wizard.png Binary files differnew file mode 100644 index 00000000..eff6ad09 --- /dev/null +++ b/docs/screenshots/win7-wizard.png diff --git a/docs/screenshots/winpythoninstaller.jpg b/docs/screenshots/winpythoninstaller.jpg Binary files differnew file mode 100644 index 00000000..0473c66a --- /dev/null +++ b/docs/screenshots/winpythoninstaller.jpg diff --git a/doc-src/dev-docs/inlinescripts.rst b/docs/scripting/inlinescripts.rst index 0c53b1a4..9b5ced5b 100644 --- a/doc-src/dev-docs/inlinescripts.rst +++ b/docs/scripting/inlinescripts.rst @@ -1,3 +1,5 @@ +.. _inline-scripts: + Inline Scripts ============== @@ -56,6 +58,7 @@ Events are listed in the order they usually occur. a connection can correspond to multiple HTTP requests. .. versionchanged:: 0.14 + :param Layer root_layer: The root layer (see :ref:`protocols` for an explanation what the root layer is), which provides transparent access to all attributes of the :py:class:`~libmproxy.proxy.RootContext`. For example, ``root_layer.client_conn.address`` @@ -111,6 +114,7 @@ Events are listed in the order they usually occur. Called when the proxy has closed the server connection. .. versionadded:: 0.14 + :param ServerConnection server_conn: see :py:func:`serverconnect` .. py:function:: clientdisconnect(context, root_layer) @@ -118,6 +122,7 @@ Events are listed in the order they usually occur. Called when a client disconnects from the proxy. .. versionchanged:: 0.14 + :param Layer root_layer: see :py:func:`clientconnect` .. py:function:: done(context) diff --git a/docs/scripting/libmproxy.rst b/docs/scripting/libmproxy.rst new file mode 100644 index 00000000..e263b89b --- /dev/null +++ b/docs/scripting/libmproxy.rst @@ -0,0 +1,27 @@ +.. _libmproxy: + +libmproxy +========= + +.. note:: + + We strongly encourage you to use :ref:`inline-scripts` rather than libmproxy. + - Inline Scripts are equally powerful and provide an easier syntax. + - Most examples are written as inline scripts. + - Multiple inline scripts can be used together. + - Inline Scripts can either be executed headless with mitmdump or within the mitmproxy UI. + + +All of mitmproxy's basic functionality is exposed through the **libmproxy** +library. The example below shows a simple implementation of the "sticky cookie" +functionality included in the interactive mitmproxy program. Traffic is +monitored for ``Cookie`` and ``Set-Cookie`` headers, and requests are rewritten +to include a previously seen cookie if they don't already have one. In effect, +this lets you log in to a site using your browser, and then make subsequent +requests using a tool like curl, which will then seem to be part of the +authenticated session. + + +.. literalinclude:: ../../examples/stickycookies + :caption: examples/stickycookies + :language: python diff --git a/docs/transparent.rst b/docs/transparent.rst new file mode 100644 index 00000000..fbc94e08 --- /dev/null +++ b/docs/transparent.rst @@ -0,0 +1,6 @@ +.. _transparent: + +Transparent Proxying +==================== + +TODO
\ No newline at end of file |
