docs :tada:

30 files changed, 872 insertions, 7 deletions

diff --git a/docs/dev/architecture.rst b/docs/dev/architecture.rst
new file mode 100644
index 00000000..ba902fab
--- /dev/null
+++ b/docs/dev/architecture.rst
@@ -0,0 +1,14 @@
+.. _architecture:
+
+Architecture
+============
+
+To give you a better understanding of how mitmproxy works, mitmproxy's
+high-level architecture is detailed in the following graphic:
+
+.. image:: ../schematics/architecture.png
+
+:download:`architecture.pdf <../schematics/architecture.pdf>`
+
+Please don't refrain from asking any further
+questions on the mailing list, the Slack channel or the GitHub issue tracker.
\ No newline at end of file
diff --git a/docs/dev/sslkeylogfile.rst b/docs/dev/sslkeylogfile.rst
new file mode 100644
index 00000000..d61a10c0
--- /dev/null
+++ b/docs/dev/sslkeylogfile.rst
@@ -0,0 +1,14 @@
+.. _sslkeylogfile:
+
+TLS Master Secrets
+==================
+
+The SSL master keys can be logged by mitmproxy so that external programs can decrypt TLS connections
+both from and to the proxy. Key logging is enabled by setting the environment variable
+:envvar:`SSLKEYLOGFILE` so that it points to a writable text file.
+Recent versions of WireShark can use these log files to decrypt packets.
+You can specify the key file path in WireShark via
+:samp:`Edit -> Preferences -> Protocols -> SSL -> (Pre)-Master-Secret log filename`.
+
+Note that :envvar:`SSLKEYLOGFILE` is respected by other programs as well, e.g. Firefox and Chrome.
+If this creates any issues, you can set :envvar:`MITMPROXY_SSLKEYLOGFILE` alternatively.
\ No newline at end of file
diff --git a/docs/dev/testing.rst b/docs/dev/testing.rst
new file mode 100644
index 00000000..242e0584
--- /dev/null
+++ b/docs/dev/testing.rst
@@ -0,0 +1,46 @@
+.. _testing:
+
+Testing
+=======
+
+All the mitmproxy projects strive to maintain 100% code coverage. In general,
+patches and pull requests will be declined unless they're accompanied by a
+suitable extension to the test suite.
+
+Our tests are written for the nose_ test framework.
+At the point where you send your pull request, a command like this:
+
+>>> nosetests --with-cov --cov-report term-missing ./test
+
+Should give output something like this:
+
+.. code-block:: none
+
+    > ---------- coverage: platform darwin, python 2.7.2-final-0 --
+    > Name                   Stmts   Miss  Cover   Missing
+    > ----------------------------------------------------
+    > libmproxy/__init__         0      0   100%
+    > libmproxy/app              4      0   100%
+    > libmproxy/cmdline        100      0   100%
+    > libmproxy/controller      69      0   100%
+    > libmproxy/dump           150      0   100%
+    > libmproxy/encoding        39      0   100%
+    > libmproxy/filt           201      0   100%
+    > libmproxy/flow           891      0   100%
+    > libmproxy/proxy          427      0   100%
+    > libmproxy/script          27      0   100%
+    > libmproxy/utils          133      0   100%
+    > libmproxy/version          4      0   100%
+    > ----------------------------------------------------
+    > TOTAL                   2045      0   100%
+    > ----------------------------------------------------
+    > Ran 251 tests in 11.864s
+
+
+There are exceptions to the coverage requirement - for instance, much of the
+console interface code can't sensibly be unit tested. These portions are
+excluded from coverage analysis either in the **.coveragerc** file, or using
+**#pragma no-cover** directives. To keep our coverage analysis relevant, we use
+these measures as sparingly as possible.
+
+.. _nose: https://nose.readthedocs.org/en/latest/
\ No newline at end of file
diff --git a/docs/features/passthrough.rst b/docs/features/passthrough.rst
new file mode 100644
index 00000000..83374955
--- /dev/null
+++ b/docs/features/passthrough.rst
@@ -0,0 +1,94 @@
+.. _passthrough:
+
+Ignore Domains
+==============
+
+There are two main reasons why you may want to exempt some traffic from mitmproxy's interception mechanism:
+
+- **Certificate pinning:** Some traffic is is protected using `Certificate Pinning`_ and
+  mitmproxy's interception leads to errors. For example, the Twitter app, Windows Update or
+  the Apple App Store fail to work if mitmproxy is active.
+- **Convenience:** You really don't care about some parts of the traffic and just want them to go away.
+
+If you want to peek into (SSL-protected) non-HTTP connections, check out the :ref:`tcpproxy` feature.
+If you want to ignore traffic from mitmproxy's processing because of large response bodies,
+take a look at the :ref:`responsestreaming` feature.
+
+How it works
+------------
+
+================== =============================
+command-line       :option:`--ignore regex`
+mitmproxy shortcut :kbd:`o` then :kbd:`I`
+================== =============================
+
+
+mitmproxy allows you to specify a regex which is matched against a ``host:port`` string
+(e.g. "example.com:443") to determine hosts that should be excluded.
+
+There are two important quirks to consider:
+
+- **In transparent mode, the ignore pattern is matched against the IP.** While we usually infer the
+  hostname from the Host header if the :option:`--host` argument is passed to mitmproxy, we do not
+  have access to this information before the SSL handshake.
+- In regular mode, explicit HTTP requests are never ignored. [#explicithttp]_ The ignore pattern is
+  applied on CONNECT requests, which initiate HTTPS or clear-text WebSocket connections.
+
+Tutorial
+--------
+
+If you just want to ignore one specific domain, there's usually a bulletproof method to do so:
+
+1. Run mitmproxy or mitmdump in verbose mode (:option:`-v`) and observe the ``host:port``
+   information in the serverconnect messages. mitmproxy will filter on these.
+2. Take the ``host:port`` string, surround it with ^ and $, escape all dots (. becomes \\.)
+   and use this as your ignore pattern:
+
+.. code-block:: none
+    :emphasize-lines: 6,7,9
+
+    >>> mitmdump -v
+    127.0.0.1:50588: clientconnect
+    127.0.0.1:50588: request
+      -> CONNECT example.com:443 HTTP/1.1
+    127.0.0.1:50588: Set new server address: example.com:443
+    127.0.0.1:50588: serverconnect
+      -> example.com:443
+    ^C
+    >>> mitmproxy --ignore ^example\.com:443$
+
+
+Here are some other examples for ignore patterns:
+
+.. code-block:: none
+
+    # Exempt traffic from the iOS App Store (the regex is lax, but usually just works):
+    --ignore apple.com:443
+    # "Correct" version without false-positives:
+    --ignore '^(.+\.)?apple\.com:443$'
+
+    # Ignore example.com, but not its subdomains:
+    --ignore '^example.com:'
+
+    # Ignore everything but example.com and mitmproxy.org:
+    --ignore '^(?!example\.com)(?!mitmproxy\.org)'
+
+    # Transparent mode:
+    --ignore 17\.178\.96\.59:443
+    # IP address range:
+    --ignore 17\.178\.\d+\.\d+:443
+
+
+.. seealso::
+
+    - :ref:`tcpproxy`
+    - :ref:`responsestreaming`
+
+.. rubric:: Footnotes
+
+.. [#explicithttp] This stems from an limitation of explicit HTTP proxying:
+    A single connection can be re-used for multiple target domains - a
+    ``GET http://example.com/`` request may be followed by a ``GET http://evil.com/`` request on the
+    same connection. If we start to ignore the connection after the first request,
+    we would miss the relevant second one.
+.. _Certificate Pinning: https://security.stackexchange.com/questions/29988/what-is-certificate-pinning
\ No newline at end of file
diff --git a/docs/features/proxyauth.rst b/docs/features/proxyauth.rst
new file mode 100644
index 00000000..edc428a7
--- /dev/null
+++ b/docs/features/proxyauth.rst
@@ -0,0 +1,17 @@
+.. _proxyauth:
+
+Proxy Authentication
+====================
+
+
+Asks the user for authentication before they are permitted to use the proxy.
+Authentication headers are stripped from the flows, so they are not passed to
+upstream servers. For now, only HTTP Basic authentication is supported. The
+proxy auth options are not compatible with the transparent, socks or reverse proxy
+mode.
+
+================== =============================
+command-line       :option:`--nonanonymous`,
+                   :option:`--singleuser USER`,
+                   :option:`--htpasswd PATH`
+================== =============================
\ No newline at end of file
diff --git a/docs/features/responsestreaming.rst b/docs/features/responsestreaming.rst
new file mode 100644
index 00000000..50fd0614
--- /dev/null
+++ b/docs/features/responsestreaming.rst
@@ -0,0 +1,68 @@
+.. _responsestreaming:
+
+Response Streaming
+==================
+
+By using mitmproxy's streaming feature, response contents can be passed to the client incrementally
+before they have been fully received by the proxy. This is especially useful for large binary files
+such as videos, where buffering the whole file slows down the client's browser.
+
+By default, mitmproxy will read the entire response, perform any indicated
+manipulations on it and then send the (possibly modified) response to
+the client. In some cases this is undesirable and you may wish to "stream"
+the reponse back to the client. When streaming is enabled, the response is
+not buffered on the proxy but directly sent back to the client instead.
+
+On the command-line
+-------------------
+
+Streaming can be enabled on the command line for all response bodies exceeding a certain size.
+The SIZE argument understands k/m/g suffixes, e.g. 3m for 3 megabytes.
+
+================== =============================
+command-line       :option:`--stream SIZE`
+================== =============================
+
+.. warning::
+
+    When response streaming is enabled, **streamed response contents will not be
+    recorded or preserved in any way.**
+
+.. note::
+
+    When response streaming is enabled, the response body cannot be modified by the usual means.
+
+Customizing Response Streaming
+------------------------------
+
+You can also use an :ref:`inlinescripts` to customize exactly
+which responses are streamed.
+
+Responses that should be tagged for streaming by setting their ``.stream`` attribute to ``True``:
+
+.. literalinclude:: ../../examples/stream.py
+   :caption: examples/stream.py
+   :language: python
+
+Implementation Details
+----------------------
+
+When response streaming is enabled, portions of the code which would have otherwise performed changes
+on the response body will see an empty response body instead (:py:data:`netlib.http.CONTENT_MISSING`).
+Any modifications will be ignored.
+
+Streamed responses are usually sent in chunks of 4096 bytes. If the response is sent with a
+``Transfer-Encoding: chunked`` header, the response will be streamed one chunk at a time.
+
+Modifying streamed data
+-----------------------
+
+If the ``.stream`` attribute is callable, ``.stream`` will wrap the generator that yields all chunks.
+
+.. literalinclude:: ../../examples/stream_modify.py
+   :caption: examples/stream_modify.py
+   :language: python
+
+.. seealso::
+
+    - :ref:`passthrough`
\ No newline at end of file
diff --git a/docs/features/reverseproxy.rst b/docs/features/reverseproxy.rst
new file mode 100644
index 00000000..87a598ff
--- /dev/null
+++ b/docs/features/reverseproxy.rst
@@ -0,0 +1,56 @@
+.. _reverseproxy:
+
+Reverse Proxy
+=============
+
+In reverse proxy mode, mitmproxy accepts standard HTTP requests and forwards
+them to the specified upstream server. This is in contrast to :ref:`upstreamproxy`, in which
+mitmproxy forwards HTTP proxy requests to an upstream proxy server.
+
+================== =====================================
+command-line       :option:`-R http[s]://hostname[:port]`
+================== =====================================
+
+Here, **scheme** signifies if the proxy should use TLS to connect to the server.
+mitmproxy always accepts both encrypted and unencrypted requests and transforms
+them to what the server expects.
+
+.. code-block:: none
+
+    >>> mitmdump -R https://httpbin.org -p 80
+    >>> curl http://localhost/
+    # requests will be transparently upgraded to TLS by mitmproxy
+
+    >>> mitmdump -R https://httpbin.org -p 443
+    >>> curl https://localhost/
+    # mitmproxy will use TLS on both ends.
+
+
+Host Header
+-----------
+
+In reverse proxy mode, mitmproxy does not rewrite the host header. While often useful, this
+may lead to issues with public web servers. For example, consider the following scenario:
+
+.. code-block:: none
+    :emphasize-lines: 5
+
+    >>> mitmdump -d -R http://example.com/
+    >>> curl http://localhost:8080/
+
+    >> GET https://example.com/
+        Host: localhost:8080
+        User-Agent: curl/7.35.0
+        [...]
+
+    << 404 Not Found 345B
+
+Since the Host header doesn't match "example.com", an error is returned.
+There are two ways to solve this:
+
+1. Modify the hosts file of your OS so that "example.com" resolves to your proxy's IP.
+   Then, access example.com directly. Make sure that your proxy can still resolve the original IP
+   or specify an IP in mitmproxy.
+2. Use mitmproxy's :ref:`setheaders` feature to rewrite the host header: ``--setheader :~q:Host:example.com``.
+   However, keep in mind that absolute URLs within the returned document or HTTP redirects will
+   cause the client application to bypass the proxy.
diff --git a/docs/features/setheaders.rst b/docs/features/setheaders.rst
index 0a6c2296..f118e6f8 100644
--- a/docs/features/setheaders.rst
+++ b/docs/features/setheaders.rst
@@ -7,7 +7,8 @@ This feature lets you specify a set of headers to be added to requests or
 responses, based on a filter pattern. You can specify these either on the
 command-line, or through an interactive editor in mitmproxy.
 
-Example:
+Example: Set the **Host** header to "example.com" for all requests.
+
 .. code-block:: none
 
     mitmdump -R http://example.com --setheader :~q:Host:example.com
diff --git a/docs/features/socksproxy.rst b/docs/features/socksproxy.rst
new file mode 100644
index 00000000..fb9117f2
--- /dev/null
+++ b/docs/features/socksproxy.rst
@@ -0,0 +1,10 @@
+.. _socksproxy:
+
+SOCKS Mode
+==========
+
+In this mode, mitmproxy acts as a SOCKS5 proxy server.
+
+================== =================
+command-line       :option:`--socks`
+================== =================
\ No newline at end of file
diff --git a/docs/features/sticky.rst b/docs/features/sticky.rst
new file mode 100644
index 00000000..e155fb9b
--- /dev/null
+++ b/docs/features/sticky.rst
@@ -0,0 +1,41 @@
+.. _sticky:
+
+Sticky cookies and auth
+=======================
+
+Sticky cookies
+--------------
+
+When the sticky cookie option is set, __mitmproxy__ will add the cookie most
+recently set by the server to any cookie-less request. Consider a service that
+sets a cookie to track the session after authentication. Using sticky cookies,
+you can fire up mitmproxy, and authenticate to a service as you usually would
+using a browser. After authentication, you can request authenticated resources
+through mitmproxy as if they were unauthenticated, because mitmproxy will
+automatically add the session tracking cookie to requests. Among other things,
+this lets you script interactions with authenticated resources (using tools
+like wget or curl) without having to worry about authentication.
+
+Sticky cookies are especially powerful when used in conjunction with :ref:`clientreplay` - you can
+record the authentication process once, and simply replay it on startup every time you need
+to interact with the secured resources.
+
+================== ======================
+command-line       :option:`-t FILTER`
+mitmproxy shortcut :kbd:`o` then :kbd:`t`
+================== ======================
+
+
+Sticky auth
+-----------
+
+The sticky auth option is analogous to the sticky cookie option, in that HTTP
+**Authorization** headers are simply replayed to the server once they have been
+seen. This is enough to allow you to access a server resource using HTTP Basic
+authentication through the proxy. Note that :program:`mitmproxy` doesn't (yet) support
+replay of HTTP Digest authentication.
+
+================== ======================
+command-line       :option:`-u FILTER`
+mitmproxy shortcut :kbd:`o` then :kbd:`A`
+================== ======================
\ No newline at end of file
diff --git a/docs/features/tcpproxy.rst b/docs/features/tcpproxy.rst
new file mode 100644
index 00000000..53df8ed6
--- /dev/null
+++ b/docs/features/tcpproxy.rst
@@ -0,0 +1,30 @@
+.. _tcpproxy:
+
+TCP Proxy
+=========
+
+WebSockets or other non-HTTP protocols are not supported by mitmproxy yet. However, you can exempt
+hostnames from processing, so that mitmproxy acts as a generic TCP forwarder.
+This feature is closely related to the :ref:`passthrough` functionality,
+but differs in two important aspects:
+
+- The raw TCP messages are printed to the event log.
+- SSL connections will be intercepted.
+
+Please note that message interception or modification are not possible yet.
+If you are not interested in the raw TCP messages, you should use the ignore domains feature.
+
+How it works
+------------
+
+================== ======================
+command-line       :option:`--tcp HOST`
+mitmproxy shortcut :kbd:`o` then :kbd:`T`
+================== ======================
+
+For a detailed description how the hostname pattern works, please look at the :ref:`passthrough` feature.
+
+.. seealso::
+
+    - :ref:`passthrough`
+    - :ref:`responsestreaming`
diff --git a/docs/features/upstreamcerts.rst b/docs/features/upstreamcerts.rst
index a287daef..84cfb84e 100644
--- a/docs/features/upstreamcerts.rst
+++ b/docs/features/upstreamcerts.rst
@@ -1,4 +1,23 @@
 .. _upstreamcerts:
 
 Upstream Certificates
-=====================
\ No newline at end of file
+=====================
+
+When mitmproxy receives a connection destined for an SSL-protected service, it
+freezes the connection before reading its request data, and makes a connection
+to the upstream server to "sniff" the contents of its SSL certificate. The
+information gained - the **Common Name** and **Subject Alternative Names** - is
+then used to generate the interception certificate, which is sent to the client
+so the connection can continue.
+
+This rather intricate little dance lets us seamlessly generate correct
+certificates even if the client has specifed only an IP address rather than the
+hostname. It also means that we don't need to sniff additional data to generate
+certs in transparent mode.
+
+Upstream cert sniffing is on by default, and can optionally be turned off.
+
+================== =============================
+command-line       :option:`--no-upstream-cert`
+mitmproxy shortcut :kbd:`o` then :kbd:`U`
+================== =============================
\ No newline at end of file
diff --git a/docs/features/upstreamproxy.rst b/docs/features/upstreamproxy.rst
new file mode 100644
index 00000000..e06833c2
--- /dev/null
+++ b/docs/features/upstreamproxy.rst
@@ -0,0 +1,12 @@
+.. _upstreamproxy:
+
+Upstream proxy mode
+===================
+
+In this mode, mitmproxy accepts proxy requests and unconditionally forwards all
+requests to a specified upstream proxy server. This is in contrast to :ref:`reverseproxy`,
+in which mitmproxy forwards ordinary HTTP requests to an upstream server.
+
+================== ===================================
+command-line       :option:`-U http://hostname[:port]`
+================== ===================================
diff --git a/docs/index.rst b/docs/index.rst
index 92583075..30fb4027 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -27,10 +27,28 @@
     features/filters
     features/replacements
     features/clientreplay
+    features/serverreplay
+    features/setheaders
+    features/passthrough
+    features/proxyauth
+    features/reverseproxy
+    features/responsestreaming
+    features/socksproxy
+    features/sticky
+    features/tcpproxy
+    features/upstreamproxy
     features/upstreamcerts
 
 .. toctree::
     :hidden:
+    :caption: Transparent Proxying
+
+    transparent
+    transparent/linux
+    transparent/osx
+
+.. toctree::
+    :hidden:
     :caption: Scripting
 
     scripting/inlinescripts
@@ -39,8 +57,19 @@
 
 .. toctree::
     :hidden:
-    :caption: Development
+    :caption: Tutorials
+
+    tutorials/30second
+    tutorials/gamecenter
+    tutorials/transparent-dhcp
+
+.. toctree::
+    :hidden:
+    :caption: Hacking
 
+    dev/architecture
+    dev/testing
+    dev/sslkeylogfile
     dev/protocols
     dev/proxy
     dev/exceptions
diff --git a/docs/modes.rst b/docs/modes.rst
index 197c0525..1be7c475 100644
--- a/docs/modes.rst
+++ b/docs/modes.rst
@@ -91,7 +91,7 @@ In this scenario, we would:
 2. Configure the client to use the proxy machine's IP as the default gateway.
 3. Quick Check: At this point, you should already be able to visit an
    unencrypted HTTP site over the proxy.
-4. Open the magic domain **mitm.it**mitm and install the certificate
+4. Open the magic domain **mitm.it** and install the certificate
    for your device.
 
 Setting the custom gateway on clients can be automated by serving the settings
diff --git a/docs/scripting/inlinescripts.rst b/docs/scripting/inlinescripts.rst
index 5914def8..19e17582 100644
--- a/docs/scripting/inlinescripts.rst
+++ b/docs/scripting/inlinescripts.rst
@@ -1,4 +1,4 @@
-.. _inline-scripts:
+.. _inlinescripts:
 
 Inline Scripts
 ==============
diff --git a/docs/scripting/libmproxy.rst b/docs/scripting/libmproxy.rst
index e263b89b..92fa5277 100644
--- a/docs/scripting/libmproxy.rst
+++ b/docs/scripting/libmproxy.rst
@@ -5,7 +5,7 @@ libmproxy
 
 .. note::
 
-    We strongly encourage you to use :ref:`inline-scripts` rather than libmproxy.
+    We strongly encourage you to use :ref:`inlinescripts` rather than libmproxy.
         - Inline Scripts are equally powerful and provide an easier syntax.
         - Most examples are written as inline scripts.
         - Multiple inline scripts can be used together.
diff --git a/docs/transparent.rst b/docs/transparent.rst
index fbc94e08..05325283 100644
--- a/docs/transparent.rst
+++ b/docs/transparent.rst
@@ -3,4 +3,22 @@
 Transparent Proxying
 ====================
 
-TODO
\ No newline at end of file
+When a transparent proxy is used, traffic is redirected into a proxy at the
+network layer, without any client configuration being required. This makes
+transparent proxying ideal for those situations where you can't change client
+behaviour - proxy-oblivious Android applications being a common example.
+
+To set up transparent proxying, we need two new components. The first is a
+redirection mechanism that transparently reroutes a TCP connection destined for
+a server on the Internet to a listening proxy server. This usually takes the
+form of a firewall on the same host as the proxy server - iptables_ on Linux
+or pf_ on OSX. When the proxy receives a redirected connection, it sees a vanilla
+HTTP request, without a host specification. This is where the second new component
+comes in - a host module that allows us to query the redirector for the original
+destination of the TCP connection.
+
+At the moment, mitmproxy supports transparent proxying on OSX Lion and above,
+and all current flavors of Linux.
+
+.. _iptables: http://www.netfilter.org/
+.. _pf: https://en.wikipedia.org/wiki/PF_\(firewall\)
\ No newline at end of file
diff --git a/docs/transparent/linux.rst b/docs/transparent/linux.rst
new file mode 100644
index 00000000..ce79128c
--- /dev/null
+++ b/docs/transparent/linux.rst
@@ -0,0 +1,45 @@
+.. _linux:
+
+Linux
+=====
+
+On Linux, mitmproxy integrates with the iptables redirection mechanism to
+achieve transparent mode.
+
+ 1. :ref:`Install the mitmproxy certificate on the test device <certinstall>`
+
+ 2. Enable IP forwarding:
+
+    >>> sysctl -w net.ipv4.ip_forward=1
+
+    You may also want to consider enabling this permanently in ``/etc/sysctl.conf``.
+
+ 3. If your target machine is on the same physical network and you configured it to use a custom
+    gateway, disable ICMP redirects:
+
+    >>> echo 0 | sudo tee /proc/sys/net/ipv4/conf/*/send_redirects
+
+    You may also want to consider enabling this permanently in ``/etc/sysctl.conf``
+    as demonstrated `here <https://unix.stackexchange.com/a/58081>`_.
+
+ 4. Create an iptables ruleset that redirects the desired traffic to the
+    mitmproxy port. Details will differ according to your setup, but the
+    ruleset should look something like this:
+
+    .. code-block:: none
+
+        iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080
+        iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -j REDIRECT --to-port 8080
+
+ 5. Fire up mitmproxy. You probably want a command like this:
+
+    >>> mitmproxy -T --host
+
+    The :option:`-T` flag turns on transparent mode, and the :option:`--host`
+    argument tells mitmproxy to use the value of the Host header for URL display.
+
+ 6. Finally, configure your test device to use the host on which mitmproxy is
+    running as the default gateway.
+
+
+For a detailed walkthrough, have a look at the :ref:`transparent-dhcp` tutorial.
diff --git a/docs/transparent/osx.rst b/docs/transparent/osx.rst
new file mode 100644
index 00000000..70adc56b
--- /dev/null
+++ b/docs/transparent/osx.rst
@@ -0,0 +1,70 @@
+.. _osx:
+
+OSX
+===
+
+OSX Lion integrated the pf_ packet filter from the OpenBSD project,
+which mitmproxy uses to implement transparent mode on OSX.
+Note that this means we don't support transparent mode for earlier versions of OSX.
+
+ 1. :ref:`Install the mitmproxy certificate on the test device <certinstall>`
+
+ 2. Enable IP forwarding:
+
+    >>> sudo sysctl -w net.inet.ip.forwarding=1
+
+ 3. Place the following two lines in a file called, say, **pf.conf**:
+
+    .. code-block:: none
+
+        rdr on en2 inet proto tcp to any port 80 -> 127.0.0.1 port 8080
+        rdr on en2 inet proto tcp to any port 443 -> 127.0.0.1 port 8080
+
+    These rules tell pf to redirect all traffic destined for port 80 or 443
+    to the local mitmproxy instance running on port 8080. You should
+    replace ``en2`` with the interface on which your test device will appear.
+
+ 4. Configure pf with the rules:
+
+    >>> sudo pfctl -f pf.conf
+
+ 5. And now enable it:
+
+    >>>sudo pfctl -e
+
+ 6. Configure sudoers to allow mitmproxy to access pfctl. Edit the file
+    **/etc/sudoers** on your system as root. Add the following line to the end
+    of the file:
+
+    .. code-block:: none
+
+        ALL ALL=NOPASSWD: /sbin/pfctl -s state
+
+    Note that this allows any user on the system to run the command
+    ``/sbin/pfctl -s state`` as root without a password. This only allows
+    inspection of the state table, so should not be an undue security risk. If
+    you're special feel free to tighten the restriction up to the user running
+    mitmproxy.
+
+ 7. Fire up mitmproxy. You probably want a command like this:
+
+    >>> mitmproxy -T --host
+
+    The :option:`-T` flag turns on transparent mode, and the :option:`--host`
+    argument tells mitmproxy to use the value of the Host header for URL display.
+
+ 8. Finally, configure your test device to use the host on which mitmproxy is
+    running as the default gateway.
+
+.. note::
+
+    Note that the **rdr** rules in the pf.conf given above only apply to inbound
+    traffic. **This means that they will NOT redirect traffic coming from the box
+    running pf itself.** We can't distinguish between an outbound connection from a
+    non-mitmproxy app, and an outbound connection from mitmproxy itself - if you
+    want to intercept your OSX traffic, you should use an external host to run
+    mitmproxy. None the less, pf is flexible to cater for a range of creative
+    possibilities, like intercepting traffic emanating from VMs.  See the
+    **pf.conf** man page for more.
+
+.. _pf: https://en.wikipedia.org/wiki/PF_\(firewall\)
\ No newline at end of file
diff --git a/docs/tutorials/30second.rst b/docs/tutorials/30second.rst
new file mode 100644
index 00000000..75304bda
--- /dev/null
+++ b/docs/tutorials/30second.rst
@@ -0,0 +1,66 @@
+.. _30second:
+
+Client playback: a 30 second example
+====================================
+
+My local cafe is serviced by a rickety and unreliable wireless network,
+generously sponsored with ratepayers' money by our city council. After
+connecting, you  are redirected to an SSL-protected page that prompts you for a
+username and password. Once you've entered your details, you are free to enjoy
+the intermittent dropouts, treacle-like speeds and incorrectly configured
+transparent proxy.
+
+I tend to automate this kind of thing at the first opportunity, on the theory
+that time spent now will be more than made up in the long run. In this case, I
+might use Firebug_ to ferret out the form post
+parameters and target URL, then fire up an editor to write a little script
+using Python's urllib_ to simulate a submission.
+That's a lot of futzing about. With mitmproxy we can do the job
+in literally 30 seconds, without having to worry about any of the details.
+Here's how.
+
+1. Run mitmdump to record our HTTP conversation to a file.
+----------------------------------------------------------
+
+>>> mitmdump -w wireless-login
+
+2. Point your browser at the mitmdump instance.
+-----------------------------------------------
+
+I use a tiny Firefox addon called `Toggle Proxy`_ to switch quickly to and from mitmproxy.
+I'm assuming you've already :ref:`configured
+your browser with mitmproxy's SSL certificate
+authority <certinstall>`.
+
+3. Log in as usual.
+-------------------
+
+And that's it! You now have a serialized version of the login process in the
+file wireless-login, and you can replay it at any time like this:
+
+>>> mitmdump -c wireless-login
+
+Embellishments
+--------------
+
+We're really done at this point, but there are a couple of embellishments we
+could make if we wanted. I use wicd_ to
+automatically join wireless networks I frequent, and it lets me specify a
+command to run after connecting. I used the client replay command above and
+voila! - totally hands-free wireless network startup.
+
+We might also want to prune requests that download CSS, JS, images and so
+forth. These add only a few moments to the time it takes to replay, but they're
+not really needed and I somehow feel compelled to trim them anyway. So, we fire up
+the mitmproxy console tool on our serialized conversation, like so:
+
+>>> mitmproxy -r wireless-login
+
+We can now go through and manually delete (using the :kbd:`d` keyboard shortcut)
+everything we want to trim. When we're done, we use :kbd:`w` to save the
+conversation back to the file.
+
+.. _Firebug: https://getfirebug.com/
+.. _urllib: https://docs.python.org/library/urllib.html
+.. _Toggle Proxy: https://addons.mozilla.org/en-us/firefox/addon/toggle-proxy-51740/
+.. _wicd: https://launchpad.net/wicd
\ No newline at end of file
diff --git a/docs/tutorials/gamecenter.rst b/docs/tutorials/gamecenter.rst
new file mode 100644
index 00000000..18745f2d
--- /dev/null
+++ b/docs/tutorials/gamecenter.rst
@@ -0,0 +1,128 @@
+.. _gamecenter:
+
+Setting highscores on Apple's GameCenter
+========================================
+
+The setup
+---------
+
+In this tutorial, I'm going to show you how simple it is to creatively
+interfere with Apple Game Center traffic using mitmproxy. To set things up,
+:ref:`install the mitmproxy root certificate <certinstall>`. Then
+start mitmproxy on your desktop, and confige the iPhone to use it as a proxy.
+
+
+Taking a look at the Game Center traffic
+----------------------------------------
+
+Lets take a first look at the Game Center traffic. The game I'll use in this
+tutorial is `Super Mega Worm`_ - a great little retro-apocalyptic sidescroller for the iPhone:
+
+.. image:: supermega.png
+    :align: center
+
+
+After finishing a game (take your time), watch the traffic flowing through
+mitmproxy:
+
+.. image:: one.png
+    :align: center
+
+We see a bunch of things we might expect - initialisation, the retrieval of
+leaderboards and so forth. Then, right at the end, there's a POST to this
+tantalising URL:
+
+.. code-block:: none
+
+    https://service.gc.apple.com/WebObjects/GKGameStatsService.woa/wa/submitScore
+
+The contents of the submission are particularly interesting:
+
+.. code-block:: xml
+
+    <!--(block|syntax("xml"))-->
+    <plist version="1.0">
+      <dict>
+        <key>scores</key>
+        <array>
+          <dict>
+            <key>category</key>
+            <string>SMW_Adv_USA1</string>
+            <key>context</key>
+            <integer>0</integer>
+            <key>score-value</key>
+            <integer>0</integer>
+            <key>timestamp</key>
+            <integer>1363515361321</integer>
+          </dict>
+        </array>
+      </dict>
+    </plist>
+    <!--(end)-->
+
+This is a `property list`_, containing an identifier for the game,
+a score (55, in this case), and a timestamp. Looks pretty simple to mess with.
+
+Modifying and replaying the score submission
+--------------------------------------------
+
+Lets edit the score submission. First, select it in mitmproxy, then press
+:kbd:`enter` to view it. Make sure you're viewing the request, not the response -
+you can use :kbd:`tab` to flick between the two. Now press :kbd:`e` for edit. You'll
+be prompted for the part of the request you want to change - press :kbd:`r` for
+raw body.  Your preferred editor (taken from the EDITOR environment variable) will
+now fire up. Lets bump the score up to something a bit more ambitious:
+
+.. code-block:: xml
+
+    <!--(block|syntax("xml"))-->
+    <plist version="1.0">
+      <dict>
+        <key>scores</key>
+        <array>
+          <dict>
+            <key>category</key>
+            <string>SMW_Adv_USA1</string>
+            <key>context</key>
+            <integer>0</integer>
+            <key>score-value</key>
+            <integer>2200272667</integer>
+            <key>timestamp</key>
+            <integer>1363515361321</integer>
+          </dict>
+        </array>
+      </dict>
+    </plist>
+    <!--(end)-->
+
+Save the file and exit your editor.
+
+The final step is to replay this modified request. Simply press :kbd:`r` for replay.
+
+
+The glorious result and some intrigue
+-------------------------------------
+
+.. image:: leaderboard.png
+    :align: center
+
+And that's it - according to the records, I am the greatest Super Mega Worm
+player of all time.
+
+There's a curious addendum to this tale. When I first wrote this tutorial, all
+the top competitors' scores were the same: 2,147,483,647 (this is no longer the
+case, beacause there are now so many fellow cheaters using this tutorial). If
+you think that number seems familiar, you're right: it's 2^31-1, the maximum
+value you can fit into a signed 32-bit int. Now let me tell you another
+peculiar thing about Super Mega Worm - at the end of every game, it submits
+your highest previous score to the Game Center, not your current score.  This
+means that it stores your highscore somewhere, and I'm guessing that it reads
+that stored score back into a signed integer. So, if you _were_ to cheat by the
+relatively pedestrian means of modifying the saved score on your jailbroken
+phone, then 2^31-1 might well be the maximum score you could get. Then again,
+if the game itself stores its score in a signed 32-bit int, you could get the
+same score through perfect play, effectively beating the game. So, which is it
+in this case? I'll leave that for you to decide.
+
+.. _Super Mega Worm: https://itunes.apple.com/us/app/super-mega-worm/id388541990?mt=8
+.. _property list: https://en.wikipedia.org/wiki/Property_list
\ No newline at end of file
diff --git a/docs/tutorials/leaderboard.png b/docs/tutorials/leaderboard.png
new file mode 100644
index 00000000..c1be8df5
--- /dev/null
+++ b/docs/tutorials/leaderboard.png
diff --git a/docs/tutorials/one.png b/docs/tutorials/one.png
new file mode 100644
index 00000000..78a636cf
--- /dev/null
+++ b/docs/tutorials/one.png
diff --git a/docs/tutorials/supermega.png b/docs/tutorials/supermega.png
new file mode 100644
index 00000000..d416f71f
--- /dev/null
+++ b/docs/tutorials/supermega.png
diff --git a/docs/tutorials/transparent-dhcp.rst b/docs/tutorials/transparent-dhcp.rst
new file mode 100644
index 00000000..1e53b333
--- /dev/null
+++ b/docs/tutorials/transparent-dhcp.rst
@@ -0,0 +1,87 @@
+.. _transparent-dhcp:
+
+Transparently proxify virtual machines
+======================================
+
+This walkthrough illustrates how to set up transparent proxying with mitmproxy.
+We use VirtualBox VMs with an Ubuntu proxy machine in this example,
+but the general *Internet <--> Proxy VM <--> (Virtual) Internal Network* setup can be applied to other setups.
+
+1. Configure Proxy VM
+---------------------
+
+On the proxy machine, **eth0** is connected to the internet. **eth1** is connected to the internal
+network that will be proxified and configured to use a static ip (192.168.3.1).
+
+VirtualBox configuration
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. image:: transparent-dhcp/step1_vbox_eth0.png
+
+.. image:: transparent-dhcp/step1_vbox_eth1.png
+
+VM Network Configuration
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. image:: transparent-dhcp/step1_proxy.png
+    :align: center
+
+2. Configure DHCP and DNS
+-------------------------
+
+We use dnsmasq to provide DHCP and DNS in our internal network.
+Dnsmasq is a lightweight server designed to provide DNS (and optionally
+DHCP and TFTP) services to a small-scale network.
+
+-   Before we get to that, we need to fix some Ubuntu quirks:
+    **Ubuntu >12.04** runs an internal dnsmasq instance (listening on loopback only) by default
+    `[1] <https://www.stgraber.org/2012/02/24/dns-in-ubuntu-12-04/>`_. For our use case, this needs to be
+    disabled by changing ``dns=dnsmasq`` to ``#dns=dnsmasq`` in **/etc/NetworkManager/NetworkManager.conf**
+    and running
+
+    >>> sudo restart network-manager
+
+    afterwards.
+-   Now, dnsmasq can be be installed and configured:
+
+    >>> sudo apt-get install dnsmasq
+
+    Replace **/etc/dnsmasq.conf** with the following configuration:
+
+    .. code-block:: none
+
+        # Listen for DNS requests on the internal network
+        interface=eth1
+        # Act as a DHCP server, assign IP addresses to clients
+        dhcp-range=192.168.3.10,192.168.3.100,96h
+        # Broadcast gateway and dns server information
+        dhcp-option=option:router,192.168.3.1
+        dhcp-option=option:dns-server,192.168.3.1
+
+    Apply changes:
+
+    >>> sudo service dnsmasq restart
+
+    Your **proxied machine** in the internal virtual network should now receive an IP address via DHCP:
+
+    .. image:: transparent-dhcp/step2_proxied_vm.png
+
+3. Redirect traffic to mitmproxy
+------------------------------------------
+
+To redirect traffic to mitmproxy, we need to add two iptables rules:
+
+.. code-block:: none
+
+    iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 80 -j REDIRECT --to-port 8080
+    iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 443 -j REDIRECT --to-port 8080
+
+4. Run mitmproxy
+----------------
+
+Finally, we can run mitmproxy in transparent mode with
+
+>>> mitmproxy -T
+
+The proxied machine cannot to leak any data outside of HTTP or DNS requests.
+If required, you can now :ref:`install the mitmproxy certificates on the proxied machine <certinstall>`.
\ No newline at end of file
diff --git a/docs/tutorials/transparent-dhcp/step1_proxy.png b/docs/tutorials/transparent-dhcp/step1_proxy.png
new file mode 100644
index 00000000..a0c94484
--- /dev/null
+++ b/docs/tutorials/transparent-dhcp/step1_proxy.png
diff --git a/docs/tutorials/transparent-dhcp/step1_vbox_eth0.png b/docs/tutorials/transparent-dhcp/step1_vbox_eth0.png
new file mode 100644
index 00000000..4b7b4e9b
--- /dev/null
+++ b/docs/tutorials/transparent-dhcp/step1_vbox_eth0.png
diff --git a/docs/tutorials/transparent-dhcp/step1_vbox_eth1.png b/docs/tutorials/transparent-dhcp/step1_vbox_eth1.png
new file mode 100644
index 00000000..b994d4cb
--- /dev/null
+++ b/docs/tutorials/transparent-dhcp/step1_vbox_eth1.png
diff --git a/docs/tutorials/transparent-dhcp/step2_proxied_vm.png b/docs/tutorials/transparent-dhcp/step2_proxied_vm.png
new file mode 100644
index 00000000..2046cc57
--- /dev/null
+++ b/docs/tutorials/transparent-dhcp/step2_proxied_vm.png