Merge remote-tracking branch 'origin/master' into pr-3448

10 files changed, 142 insertions, 27 deletions

diff --git a/docs/src/content/_index.md b/docs/src/content/_index.md
index cd368df0..6283343d 100644
--- a/docs/src/content/_index.md
+++ b/docs/src/content/_index.md
@@ -11,8 +11,7 @@ menu:
 The mitmproxy project's tools are a set of front-ends that expose common
 underlying functionality.
 
-**mitmproxy** is an interactive man-in-the-middle proxy for HTTP and HTTPS
-with a console interface.
+**mitmproxy** is an interactive, SSL/TLS-capable intercepting proxy with a console interface for HTTP/1, HTTP/2, and WebSockets.
 
 **mitmdump** is the command-line version of mitmproxy. Think tcpdump for HTTP.
 
@@ -21,6 +20,9 @@ with a console interface.
 Documentation, tutorials and distribution packages can be found on the
 [mitmproxy website](https://mitmproxy.org).
 
+Development information and our source code can be found in our
+[GitHub repository](https://github.com/mitmproxy/mitmproxy).
+
 
 ## Features
 
@@ -29,8 +31,7 @@ Documentation, tutorials and distribution packages can be found on the
 - Replay the client-side of an HTTP conversations
 - Replay HTTP responses of a previously recorded server
 - Reverse proxy mode to forward traffic to a specified server
-- Transparent proxy mode on OSX and Linux
+- Transparent proxy mode on macOS and Linux
 - Make scripted changes to HTTP traffic using Python
 - SSL/TLS certificates for interception are generated on the fly
 - And much, much more...
-
diff --git a/docs/src/content/addons-scripting.md b/docs/src/content/addons-scripting.md
index 4e9916ca..6a18eaf4 100644
--- a/docs/src/content/addons-scripting.md
+++ b/docs/src/content/addons-scripting.md
@@ -27,6 +27,6 @@ You can look at the [http][] module, or the [Request][], and
 [Response][] classes for other attributes that you can use when
 scripting.
 
-[http][]: https://github.com/mitmproxy/mitmproxy/blob/master/mitmproxy/http.py
+[http]: https://github.com/mitmproxy/mitmproxy/blob/master/mitmproxy/http.py
 [Request]: https://github.com/mitmproxy/mitmproxy/blob/master/mitmproxy/net/http/request.py
 [Response]: https://github.com/mitmproxy/mitmproxy/blob/master/mitmproxy/net/http/response.py
diff --git a/docs/src/content/concepts-certificates.md b/docs/src/content/concepts-certificates.md
index 88482047..4e2ae47a 100644
--- a/docs/src/content/concepts-certificates.md
+++ b/docs/src/content/concepts-certificates.md
@@ -24,6 +24,9 @@ something like this:
 Click on the relevant icon, follow the setup instructions for the platform
 you're on and you are good to go.
 
+Note: If you are using an iOS device, you should be using the Safari browser
+so that it opens the proper prompts for installing the certificate.
+
 ## Installing the mitmproxy CA certificate manually
 
 Sometimes using the quick install app is not an option - Java or the iOS
diff --git a/docs/src/content/concepts-protocols.md b/docs/src/content/concepts-protocols.md
index fc056545..c79274bf 100644
--- a/docs/src/content/concepts-protocols.md
+++ b/docs/src/content/concepts-protocols.md
@@ -36,7 +36,7 @@ mitmproxy currently does not support HTTP/2 Cleartext (h2c) since none of the
 major browser vendors have implemented it.
 
 Some websites are still having problems with correct HTTP/2 support in their
-webservers and can cause errors, dropped connectiones, or simply no response at
+webservers and can cause errors, dropped connections, or simply no response at
 all. We are trying to be as tolerant and forgiving as possible with the types of
 data we send and receive, but
 [some](https://github.com/mitmproxy/mitmproxy/issues/1745)
diff --git a/docs/src/content/howto-ignoredomains.md b/docs/src/content/howto-ignoredomains.md
index 902a17be..9a337eba 100644
--- a/docs/src/content/howto-ignoredomains.md
+++ b/docs/src/content/howto-ignoredomains.md
@@ -10,7 +10,7 @@ menu:
 There are two main reasons why you may want to exempt some traffic from
 mitmproxy's interception mechanism:
 
-- **Certificate pinning:** Some traffic is is protected using [Certificate
+- **Certificate pinning:** Some traffic is protected using [Certificate
   Pinning](https://security.stackexchange.com/questions/29988/what-is-certificate-pinning)
   and mitmproxy's interception leads to errors. For example, the Twitter app,
   Windows Update or the Apple App Store fail to work if mitmproxy is active.
diff --git a/docs/src/content/howto-install-system-trusted-ca-android.md b/docs/src/content/howto-install-system-trusted-ca-android.md
new file mode 100644
index 00000000..2ef67f30
--- /dev/null
+++ b/docs/src/content/howto-install-system-trusted-ca-android.md
@@ -0,0 +1,86 @@
+---
+title: "Install System CA on Android"
+menu:
+    howto:
+        weight: 4
+---
+
+# Install System CA Certificate on Android Emulator
+
+[Since Android 7, apps ignore user certificates](https://android-developers.googleblog.com/2016/07/changes-to-trusted-certificate.html), unless they are configured to use them.
+As most applications do not explicitly opt in to use user certificates, we need to place our mitmproxy CA certificate in the system certificate store,
+in order to avid having to patch each application, which we want to monitor.
+
+Please note, that apps can decide to ignore the system certificate store and maintain their own CA certificates. In this case you have to patch the application.
+
+## 1. Prerequisites
+
+  - Emulator from Android SDK with proxy settings pointing to mitmproxy
+
+  - Mitmproxy CA certificate
+    - Usually located in `~/.mitmproxy/mitmproxy-ca-cert.cer`
+    - If the folder is empty or does not exist, run `mitmproxy` in order to generate the certificates
+    
+## 2. Rename certificate
+Enter your certificate folder
+{{< highlight bash  >}}
+cd ~/.mitmproxy/
+{{< / highlight >}}
+
+  - CA Certificates in Android are stored by the name of their hash, with a '0' as extension
+  - Now generate the hash of your certificate
+  
+{{< highlight bash  >}}
+openssl x509 -inform PEM -subject_hash_old -in mitmproxy-ca-cert.cer | head -1
+{{< / highlight >}}
+Lets assume, the output is `c8450d0d`
+
+We can now copy `mitmproxy-ca-cert.cer` to `c8450d0d.0` and our system certificate is ready to use
+{{< highlight bash  >}}
+cp mitmproxy-ca-cert.cer c8450d0d.0
+{{< / highlight >}}
+
+## 3. Insert certificate into system certificate store
+
+Note, that Android 9 (API LEVEL 28) was used to test the following steps and that the `emulator` executable is located in the Android SDK
+
+  - Start your android emulator. 
+     - Get a list of your AVDs with `emulator -list-avds`
+     - Make sure to use the `-writable-system` option. Otherwise it will not be possible to write to `/system`
+     - Keep in mind, that the **emulator will load a clean system image when starting without `-writable-system` option**.
+     - This means you always have to start the emulator with `-writable-system` option in order to use your certificate
+
+{{< highlight bash  >}}
+emulator -avd <avd_name_here> -writable-system
+{{< / highlight >}}
+
+  - Restart adb as root
+  
+{{< highlight bash  >}}
+adb root
+{{< / highlight >}}
+
+  - Get write access to `/system` on the device
+  - In earlier versions (API LEVEL < 28) of Android you have to use `adb shell "mount -o rw,remount /system"`
+  
+{{< highlight bash  >}}
+adb shell "mount -o rw,remount /"
+{{< / highlight >}}
+
+  - Push your certificate to the system certificate store and set file permissions
+  
+{{< highlight bash  >}}
+adb push c8450d0d.0 /system/etc/security/cacerts
+adb shell "chmod 664 /system/etc/security/cacerts/c8450d0d.0"
+{{< / highlight >}}
+
+## 4. Reboot device and enjoy decrypted TLS traffic
+
+  - Reboot your device. 
+     - You CA certificate should now be system trusted
+         
+{{< highlight bash  >}}
+adb reboot
+{{< / highlight >}}
+
+**Remember**: You **always** have to start the emulator using the `-writable-system` option in order to use your certificate
\ No newline at end of file
diff --git a/docs/src/content/howto-transparent-vms.md b/docs/src/content/howto-transparent-vms.md
index 1446ede7..f251bc44 100644
--- a/docs/src/content/howto-transparent-vms.md
+++ b/docs/src/content/howto-transparent-vms.md
@@ -14,9 +14,13 @@ Internal Network* setup can be applied to other setups.
 
 ## 1. Configure Proxy VM
 
-On the proxy machine, **eth0** is connected to the internet. **eth1** is
-connected to the internal network that will be proxified and configured
-to use a static ip (192.168.3.1).
+First, we have to find out under which name Ubuntu has mapped our network interfaces. You can find this information with:
+
+{{< highlight bash  >}}
+ip link
+{{< / highlight >}}
+
+Usually with Ubuntu and Virtualbox, **eth0** or **enp0s3** (Ubuntu 15.10 and newer) is connected to the internet and **eth1**  or **enp0s8** (Ubuntu 15.10 and newer) is connected to the internal network that will be proxified and configured to use a static ip (192.168.3.1). If the names differ, use the ones you got from the *ip link* command.
 
 ### VirtualBox configuration
 
@@ -65,6 +69,7 @@ Replace **/etc/dnsmasq.conf** with the following configuration:
 {{< highlight none  >}}
 # Listen for DNS requests on the internal network
 interface=eth1
+bind-interfaces
 # Act as a DHCP server, assign IP addresses to clients
 dhcp-range=192.168.3.10,192.168.3.100,96h
 # Broadcast gateway and dns server information
@@ -93,10 +98,11 @@ IP address via DHCP:
 
 ## 3. Redirect traffic to mitmproxy
 
-To redirect traffic to mitmproxy, we need to add two iptables
+To redirect traffic to mitmproxy, we need to enable IP forwarding and add two iptables
 rules:
 
 {{< highlight bash  >}}
+sudo sysctl -w net.ipv4.ip_forward=1
 sudo iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 80 -j REDIRECT --to-port 8080
 sudo iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 443 -j REDIRECT --to-port 8080
 {{< / highlight >}}
diff --git a/docs/src/content/howto-transparent.md b/docs/src/content/howto-transparent.md
index ae36f579..d448bd82 100644
--- a/docs/src/content/howto-transparent.md
+++ b/docs/src/content/howto-transparent.md
@@ -50,7 +50,7 @@ a newly created `/etc/sysctl.d/mitmproxy.conf` (see [here](https://superuser.com
 sysctl -w net.ipv4.conf.all.send_redirects=0
 {{< / highlight >}}
 
-If your test device is on the same physical network, your machine shouldn't inform the device that 
+If your test device is on the same physical network, your machine shouldn't inform the device that
 there's a shorter route available by skipping the proxy.
 
 If you want to persist this across reboots, see above.
@@ -83,9 +83,34 @@ The `--mode transparent` option turns on transparent mode, and the `--showhost`
 
 ### 5. Finally, configure your test device.
 
-Set the test device up to use the host on which mitmproxy is running as the default gateway and 
+Set the test device up to use the host on which mitmproxy is running as the default gateway and
 [install the mitmproxy certificate authority on the test device]({{< relref "concepts-certificates" >}}).
 
+### Work-around to redirect traffic originating from the machine itself
+
+Follow steps **1, 2** as above, but *instead* of the commands in step **3**, run the following
+
+Create a user to run the mitmproxy
+
+{{< highlight bash  >}}
+sudo useradd --create-home mitmproxyuser
+sudo -u mitmproxyuser bash -c 'cd ~ && pip install --user mitmproxy'
+{{< / highlight >}}
+
+Then, configure the iptables rules to redirect all traffic from our local machine to mitmproxy. **Note**, as soon as you run these, you won't be able to perform successful network calls *until* you start mitmproxy. If you run into issues, `iptables -t nat -F` is a heavy handed way to flush (clear) *all* the rules from the iptables `nat` table (which includes any other rules you had configured).
+
+{{< highlight bash  >}}
+iptables -t nat -A OUTPUT -p tcp -m owner ! --uid-owner mitmproxyuser --dport 80 -j REDIRECT --to-port 8080
+iptables -t nat -A OUTPUT -p tcp -m owner ! --uid-owner mitmproxyuser --dport 443 -j REDIRECT --to-port 8080
+ip6tables -t nat -A OUTPUT -p tcp -m owner ! --uid-owner mitmproxyuser --dport 80 -j REDIRECT --to-port 8080
+ip6tables -t nat -A OUTPUT -p tcp -m owner ! --uid-owner mitmproxyuser --dport 443 -j REDIRECT --to-port 8080
+{{< / highlight >}}
+
+This will redirect the packets from all users other than `mitmproxyuser` on the machine to mitmproxy. To avoid circularity, run mitmproxy as the user `mitmproxyuser`. Hence step **4** should look like:
+{{< highlight bash  >}}
+sudo -u mitmproxyuser bash -c '$HOME/.local/bin/mitmproxy --mode transparent --showhost --set block_global=false'
+{{< / highlight >}}
+
 
 
 ## OpenBSD
@@ -124,7 +149,7 @@ doas pfctl -e
 You probably want a command like this:
 
 {{< highlight bash  >}}
-mitmproxy --mode transparent --showhost
+mitmproxy --mode transparent --listen-host 127.0.0.1 --showhost
 {{< / highlight >}}
 
 The `--mode transparent` option turns on transparent mode, and the `--showhost` argument tells
@@ -132,7 +157,7 @@ mitmproxy to use the value of the Host header for URL display.
 
 ### 6. Finally, configure your test device.
 
-Set the test device up to use the host on which mitmproxy is running as the default gateway and 
+Set the test device up to use the host on which mitmproxy is running as the default gateway and
 [install the mitmproxy certificate authority on the test device]({{< relref "concepts-certificates" >}}).
 
 
@@ -213,7 +238,7 @@ mitmproxy to use the value of the Host header for URL display.
 
 ### 7. Finally, configure your test device.
 
-Set the test device up to use the host on which mitmproxy is running as the default gateway and 
+Set the test device up to use the host on which mitmproxy is running as the default gateway and
 [install the mitmproxy certificate authority on the test device]({{< relref "concepts-certificates" >}}).
 
 {{% note %}}
@@ -229,7 +254,7 @@ for more.
 
 ### Work-around to redirect traffic originating from the machine itself
 
-Follow the steps **1, 2** as above. In step **3** change the contents of the file **pf.conf** to
+Follow steps **1, 2** as above, but in step **2** change the contents of the file **pf.conf** to
 
 {{< highlight none >}}
 #The ports to redirect to proxy
@@ -246,18 +271,12 @@ tproxy_user = "nobody"
 #This cannot involve the user which runs the
 #transparent proxy as that would cause an infinite loop.
 #
-#Here we redirect for all users which don't run transparent proxy.
-redir_users = "{ !=" $tproxy_user "}"
-
-#If you only wish to redirect traffic for particular users
-#you may also do:
-#redir_users = "{= john, = jane}"
 
 rdr pass proto tcp from any to any port $redir_ports -> $tproxy
-pass out route-to (lo0 127.0.0.1) proto tcp from any to any port $redir_ports user $redir_users
+pass out route-to (lo0 127.0.0.1) proto tcp from any to any port $redir_ports user { != $tproxy_user }
 {{< / highlight >}}
 
-Follow steps **4-6** above. This will redirect the packets from all users other than `nobody` on the machine to mitmproxy. To avoid circularity, run mitmproxy as the user `nobody`. Hence step **7** should look like:
+Follow steps **3-5** above. This will redirect the packets from all users other than `nobody` on the machine to mitmproxy. To avoid circularity, run mitmproxy as the user `nobody`. Hence step **6** should look like:
 
 {{< highlight bash  >}}
 sudo -u nobody mitmproxy --mode transparent --showhost
diff --git a/docs/src/content/howto-wireshark-tls.md b/docs/src/content/howto-wireshark-tls.md
index 588223ac..a55d177b 100644
--- a/docs/src/content/howto-wireshark-tls.md
+++ b/docs/src/content/howto-wireshark-tls.md
@@ -7,7 +7,7 @@ menu:
 
 # Wireshark and SSL/TLS Master Secrets
 
-The SSL/SSL master keys can be logged by mitmproxy so that external programs can
+The SSL/TLS master keys can be logged by mitmproxy so that external programs can
 decrypt SSL/TLS connections both from and to the proxy. Recent versions of
 Wireshark can use these log files to decrypt packets. See the [Wireshark wiki](https://wiki.wireshark.org/SSL#Using_the_.28Pre.29-Master-Secret) for more information.
 
diff --git a/docs/src/content/tute-highscores.md b/docs/src/content/tute-highscores.md
index f5cbd7bc..2d03076d 100644
--- a/docs/src/content/tute-highscores.md
+++ b/docs/src/content/tute-highscores.md
@@ -67,7 +67,7 @@ timestamp. Looks pretty simple to mess with.
 
 Lets edit the score submission. First, select it in mitmproxy, then
 press <span data-role="kbd">enter</span> to view it. Make sure you're
-viewing the request, not the response -you can use
+viewing the request, not the response - you can use
 <span data-role="kbd">tab</span> to flick between the two. Now press
 <span data-role="kbd">e</span> for edit. You'll be prompted for the part
 of the request you want to change - press <span data-role="kbd">r</span>