diff options
75 files changed, 1601 insertions, 1130 deletions
@@ -1,6 +0,0 @@ -DIR="$( dirname "${BASH_SOURCE[0]}" )" -ACTIVATE_DIR="$(if [ -f "$DIR/venv/bin/activate" ]; then echo 'bin'; else echo 'Scripts'; fi;)" -if [ -z "$VIRTUAL_ENV" ] && [ -f "$DIR/venv/$ACTIVATE_DIR/activate" ]; then - echo "Activating mitmproxy virtualenv..." - source "$DIR/venv/$ACTIVATE_DIR/activate" -fi @@ -37,7 +37,7 @@ each other solve problems, and come up with new ideas for the project. |mitmproxy_discourse| -Join our developer chat on Slack if you would like to hack on mitmproxy itself. +Join our developer chat on Slack if you would like to contribute to mitmproxy itself. |slack| @@ -48,73 +48,64 @@ Installation The installation instructions are `here <http://docs.mitmproxy.org/en/stable/install.html>`__. If you want to contribute changes, keep on reading. +Contributing +------------ -Hacking -------- +As an open source project, mitmproxy welcomes contributions of all forms. If you would like to bring the project forward, +please consider contributing in the following areas: + +- **Maintenance:** We are *incredibly* thankful for individuals who are stepping up and helping with maintenance. This includes (but is not limited to) triaging issues, reviewing pull requests and picking up stale ones, helping out other users in our forums_, creating minimal, complete and verifiable examples or test cases for existing bug reports, updating documentation, or fixing minor bugs that have recently been reported. +- **Code Contributions:** We actively mark issues that we consider are `good first contributions`_. If you intend to work on a larger contribution to the project, please come talk to us first. + +Development Setup +----------------- -To get started hacking on mitmproxy, make sure you have Python_ 3.5.x or above with -virtualenv_ installed (you can find installation instructions for virtualenv -`here <http://virtualenv.readthedocs.org/en/latest/>`__). Then do the following: +To get started hacking on mitmproxy, please follow the `advanced installation`_ steps to install mitmproxy from source, but stop right before running ``pip3 install mitmproxy``. Instead, do the following: .. code-block:: text - sudo apt-get install python3-pip python3-dev libffi-dev libssl-dev libtiff5-dev libjpeg8-dev zlib1g-dev libwebp-dev git clone https://github.com/mitmproxy/mitmproxy.git cd mitmproxy - ./dev.sh # powershell .\dev.ps1 on Windows + ./dev.sh # "powershell .\dev.ps1" on Windows -The *dev* script will create a virtualenv environment in a directory called -"venv3.5", and install all mandatory and optional dependencies into it. The -primary mitmproxy components - mitmproxy and pathod - are installed as +The *dev* script will create a `virtualenv`_ environment in a directory called "venv" +and install all mandatory and optional dependencies into it. The primary +mitmproxy components - mitmproxy and pathod - are installed as "editable", so any changes to the source in the repository will be reflected live in the virtualenv. -To confirm that you're up and running, activate the virtualenv, and run the -mitmproxy test suite: - -.. code-block:: text - - . venv3.5/bin/activate # venv\Scripts\activate on Windows - py.test - -Note that the main executables for the project - ``mitmdump``, ``mitmproxy``, +The main executables for the project - ``mitmdump``, ``mitmproxy``, ``mitmweb``, ``pathod``, and ``pathoc`` - are all created within the virtualenv. After activating the virtualenv, they will be on your $PATH, and you can run them like any other command: .. code-block:: text + . venv/bin/activate # "venv\Scripts\activate" on Windows mitmdump --version -For convenience, the project includes an autoenv_ file (`.env`_) that -auto-activates the virtualenv when you cd into the mitmproxy directory. - - Testing ------- If you've followed the procedure above, you already have all the development -requirements installed, and you can simply run the test suite: +requirements installed, and you can run the full test suite (including tests for code style and documentation) with tox_: .. code-block:: text - py.test - -Please ensure that all patches are accompanied by matching changes in the test -suite. The project tries to maintain 100% test coverage. + tox -You can also use `tox` to run the full suite of tests, including a quick test -to check documentation and code linting. - -The following tox environments are relevant for local testing: +For speedier testing, we recommend you run `pytest`_ directly on individual test files or folders: .. code-block:: text - tox -e py35 # runs all tests with Python 3.5 - tox -e docs # runs a does-it-compile check on the documentation - tox -e lint # runs the linter for coding style checks + cd test/mitmproxy/addons + pytest --cov mitmproxy.addons.anticache --looponfail test_anticache.py + +As pytest does not check the code style, you probably want to run ``tox -e lint`` before committing your changes. +Please ensure that all patches are accompanied by matching changes in the test +suite. The project tries to maintain 100% test coverage and enforces this strictly for some parts of the codebase. Documentation ------------- @@ -133,8 +124,8 @@ installation, you can render the documentation like this: The last command invokes `sphinx-autobuild`_, which watches the Sphinx directory and rebuilds the documentation when a change is detected. -Style ------ +Code Style +---------- Keeping to a consistent code style throughout the project makes it easier to contribute and collaborate. Please stick to the guidelines in @@ -186,11 +177,13 @@ with the following command: :target: https://pypi.python.org/pypi/mitmproxy :alt: Supported Python versions -.. _Python: https://www.python.org/ -.. _virtualenv: http://virtualenv.readthedocs.org/en/latest/ -.. _autoenv: https://github.com/kennethreitz/autoenv -.. _.env: https://github.com/mitmproxy/mitmproxy/blob/master/.env +.. _`advanced installation`: http://docs.mitmproxy.org/en/latest/install.html#advanced-installation +.. _virtualenv: https://virtualenv.pypa.io/ +.. _`pytest`: http://pytest.org/ +.. _tox: https://tox.readthedocs.io/ .. _Sphinx: http://sphinx-doc.org/ .. _sphinx-autobuild: https://pypi.python.org/pypi/sphinx-autobuild .. _PEP8: https://www.python.org/dev/peps/pep-0008 -.. _Google Style Guide: https://google.github.io/styleguide/pyguide.html +.. _`Google Style Guide`: https://google.github.io/styleguide/pyguide.html +.. _forums: https://discourse.mitmproxy.org/ +.. _`good first contributions`: https://github.com/mitmproxy/mitmproxy/issues?q=is%3Aissue+is%3Aopen+label%3Agood-first-contribution @@ -1,20 +1,19 @@ $ErrorActionPreference = "Stop" -$VENV = ".\venv" $pyver = python --version if($pyver -notmatch "3\.[5-9]") { Write-Warning "Unexpected Python version, expected Python 3.5 or above: $pyver" } -python -m venv $VENV --copies -& $VENV\Scripts\activate.ps1 +python -m venv .\venv --copies +& .\venv\Scripts\activate.ps1 python -m pip install --disable-pip-version-check -U pip cmd /c "pip install -r requirements.txt 2>&1" echo @" - * Created virtualenv environment in $VENV. + * Created virtualenv environment in .\venv. * Installed all dependencies into the virtualenv. * Activated virtualenv environment. @@ -2,16 +2,14 @@ set -e set -x -PYVERSION=${1:-3.5} -VENV="venv$PYVERSION" +echo "Creating dev environment in ./venv..." -echo "Creating dev environment in $VENV using Python $PYVERSION" - -python$PYVERSION -m venv "$VENV" -. "$VENV/bin/activate" -pip$PYVERSION install -U pip setuptools -pip$PYVERSION install -r requirements.txt +python3 -m venv venv +. venv/bin/activate +pip3 install -U pip setuptools +pip3 install -r requirements.txt echo "" -echo "* Virtualenv created in $VENV and all dependencies installed." -echo "* You can now activate the $(python --version) virtualenv with this command: \`. $VENV/bin/activate\`" +echo " * Created virtualenv environment in ./venv." +echo " * Installed all dependencies into the virtualenv." +echo " * You can now activate the $(python3 --version) virtualenv with this command: \`. venv/bin/activate\`"
\ No newline at end of file diff --git a/docs/dev/architecture.rst b/docs/dev/architecture.rst deleted file mode 100644 index e7995141..00000000 --- a/docs/dev/architecture.rst +++ /dev/null @@ -1,14 +0,0 @@ -.. _architecture: - -Architecture -============ - -To give you a better understanding of how mitmproxy works, mitmproxy's -high-level architecture is detailed in the following graphic: - -.. image:: ../schematics/architecture.png - -:download:`architecture.pdf <../schematics/architecture.pdf>` - -Please don't refrain from asking any further -questions on the mailing list, the Slack channel or the GitHub issue tracker. diff --git a/docs/dev/contributing.rst b/docs/dev/contributing.rst new file mode 100644 index 00000000..8e04c270 --- /dev/null +++ b/docs/dev/contributing.rst @@ -0,0 +1,11 @@ +.. _contributing: + +Contributing +============ + +As an open source project, **mitmproxy** welcomes contributions of all forms. + +Please head over to the README_ to get started! 😃 + + +.. _README: https://github.com/mitmproxy/mitmproxy/blob/master/README.rst
\ No newline at end of file diff --git a/docs/dev/testing.rst b/docs/dev/testing.rst deleted file mode 100644 index b86efce1..00000000 --- a/docs/dev/testing.rst +++ /dev/null @@ -1,47 +0,0 @@ -.. _testing: - -Testing -======= - -All the mitmproxy projects strive to maintain 100% code coverage. In general, -patches and pull requests will be declined unless they're accompanied by a -suitable extension to the test suite. - -Our tests are written for the `py.test`_ or nose_ test frameworks. -At the point where you send your pull request, a command like this: - ->>> py.test --cov mitmproxy - -Should give output something like this: - -.. code-block:: none - - > ---------- coverage: platform darwin, python 2.7.2-final-0 -- - > Name Stmts Miss Cover Missing - > ---------------------------------------------------- - > mitmproxy/__init__ 0 0 100% - > mitmproxy/app 4 0 100% - > mitmproxy/cmdline 100 0 100% - > mitmproxy/controller 69 0 100% - > mitmproxy/dump 150 0 100% - > mitmproxy/encoding 39 0 100% - > mitmproxy/flowfilter 201 0 100% - > mitmproxy/flow 891 0 100% - > mitmproxy/proxy 427 0 100% - > mitmproxy/script 27 0 100% - > mitmproxy/utils 133 0 100% - > mitmproxy/version 4 0 100% - > ---------------------------------------------------- - > TOTAL 2045 0 100% - > ---------------------------------------------------- - > Ran 251 tests in 11.864s - - -There are exceptions to the coverage requirement - for instance, much of the -console interface code can't sensibly be unit tested. These portions are -excluded from coverage analysis either in the **.coveragerc** file, or using -**#pragma no-cover** directives. To keep our coverage analysis relevant, we use -these measures as sparingly as possible. - -.. _nose: https://nose.readthedocs.org/en/latest/ -.. _py.test: https://pytest.org/ diff --git a/docs/index.rst b/docs/index.rst index a40a5f62..8ba14f54 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -78,10 +78,9 @@ .. toctree:: :hidden: - :caption: Hacking + :caption: Development - dev/architecture - dev/testing + dev/contributing dev/sslkeylogfile .. Indices and tables diff --git a/docs/install.rst b/docs/install.rst index bcf07023..b9524897 100644 --- a/docs/install.rst +++ b/docs/install.rst @@ -20,7 +20,7 @@ You can use Homebrew to install everything: brew install mitmproxy -Or you can download the pre-built binary packages from `mitmproxy.org`_. +Or you can download the pre-built binary packages from our `releases`_. .. _install-windows: @@ -44,7 +44,7 @@ Installation on Linux --------------------- The recommended way to run mitmproxy on Linux is to use the pre-built binaries -provided at `mitmproxy.org`_. +provided at `releases`_. Our pre-built binaries provide you with the latest version of mitmproxy, a self-contained Python 3.5 environment and a recent version of OpenSSL that @@ -139,11 +139,12 @@ Latest Development Version If you would like to install mitmproxy directly from the master branch on GitHub or would like to get set up to contribute to the project, install the dependencies as you would for a regular installation from source. Then see the -Hacking_ section of the README on GitHub. You can check your system information +project's README_ on GitHub. You can check your system information by running: ``mitmproxy --version`` -.. _Hacking: https://github.com/mitmproxy/mitmproxy/blob/master/README.rst#hacking +.. _README: https://github.com/mitmproxy/mitmproxy/blob/master/README.rst +.. _releases: https://github.com/mitmproxy/mitmproxy/releases .. _mitmproxy.org: https://mitmproxy.org/ .. _`Python website`: https://www.python.org/downloads/windows/ .. _pip: https://pip.pypa.io/en/latest/installing.html diff --git a/docs/schematics/architecture.pdf b/docs/schematics/architecture.pdf Binary files differdeleted file mode 100644 index 77f5ad58..00000000 --- a/docs/schematics/architecture.pdf +++ /dev/null diff --git a/docs/schematics/architecture.png b/docs/schematics/architecture.png Binary files differdeleted file mode 100644 index 67d6c718..00000000 --- a/docs/schematics/architecture.png +++ /dev/null diff --git a/docs/schematics/architecture.vsdx b/docs/schematics/architecture.vsdx Binary files differdeleted file mode 100644 index c4ff13d2..00000000 --- a/docs/schematics/architecture.vsdx +++ /dev/null diff --git a/docs/transparent.rst b/docs/transparent.rst index 8befbac5..889079af 100644 --- a/docs/transparent.rst +++ b/docs/transparent.rst @@ -44,7 +44,7 @@ This mode does require root privileges though. There's a wrapper in the examples called 'mitmproxy_shim.c', which will enable you to use this mode with dropped priviliges. It can be used as follows:: - gcc examples/mitmproxy_shim.c -o mitmproxy_shim -lcap + gcc examples/complex/full_transparency_shim.c -o mitmproxy_shim -lcap sudo chown root:root mitmproxy_shim sudo chmod u+s mitmproxy_shim ./mitmproxy_shim $(which mitmproxy) -T --spoof-source-address diff --git a/mitmproxy/addons/__init__.py b/mitmproxy/addons/__init__.py index 8a2f2974..97fa2dcd 100644 --- a/mitmproxy/addons/__init__.py +++ b/mitmproxy/addons/__init__.py @@ -1,35 +1,40 @@ from mitmproxy.addons import anticache from mitmproxy.addons import anticomp +from mitmproxy.addons import check_alpn +from mitmproxy.addons import check_ca from mitmproxy.addons import clientplayback -from mitmproxy.addons import streamfile +from mitmproxy.addons import disable_h2c_upgrade from mitmproxy.addons import onboarding from mitmproxy.addons import proxyauth from mitmproxy.addons import replace from mitmproxy.addons import script -from mitmproxy.addons import setheaders from mitmproxy.addons import serverplayback +from mitmproxy.addons import setheaders from mitmproxy.addons import stickyauth from mitmproxy.addons import stickycookie from mitmproxy.addons import streambodies +from mitmproxy.addons import streamfile from mitmproxy.addons import upstream_auth -from mitmproxy.addons import disable_h2c_upgrade def default_addons(): return [ - onboarding.Onboarding(), - proxyauth.ProxyAuth(), anticache.AntiCache(), anticomp.AntiComp(), + check_alpn.CheckALPN(), + check_ca.CheckCA(), + clientplayback.ClientPlayback(), + disable_h2c_upgrade.DisableH2CleartextUpgrade(), + onboarding.Onboarding(), + proxyauth.ProxyAuth(), + replace.Replace(), + replace.ReplaceFile(), + script.ScriptLoader(), + serverplayback.ServerPlayback(), + setheaders.SetHeaders(), stickyauth.StickyAuth(), stickycookie.StickyCookie(), - script.ScriptLoader(), - streamfile.StreamFile(), streambodies.StreamBodies(), - replace.Replace(), - setheaders.SetHeaders(), - serverplayback.ServerPlayback(), - clientplayback.ClientPlayback(), + streamfile.StreamFile(), upstream_auth.UpstreamAuth(), - disable_h2c_upgrade.DisableH2CleartextUpgrade(), ] diff --git a/mitmproxy/addons/check_alpn.py b/mitmproxy/addons/check_alpn.py new file mode 100644 index 00000000..c288d788 --- /dev/null +++ b/mitmproxy/addons/check_alpn.py @@ -0,0 +1,17 @@ +import mitmproxy +from mitmproxy.net import tcp + + +class CheckALPN: + def __init__(self): + self.failed = False + + def configure(self, options, updated): + self.failed = mitmproxy.ctx.master.options.http2 and not tcp.HAS_ALPN + if self.failed: + mitmproxy.ctx.master.add_log( + "HTTP/2 is disabled because ALPN support missing!\n" + "OpenSSL 1.0.2+ required to support HTTP/2 connections.\n" + "Use --no-http2 to silence this warning.", + "warn", + ) diff --git a/mitmproxy/addons/check_ca.py b/mitmproxy/addons/check_ca.py new file mode 100644 index 00000000..a83ab8e1 --- /dev/null +++ b/mitmproxy/addons/check_ca.py @@ -0,0 +1,24 @@ +import mitmproxy + + +class CheckCA: + def __init__(self): + self.failed = False + + def configure(self, options, updated): + has_ca = ( + mitmproxy.ctx.master.server and + mitmproxy.ctx.master.server.config and + mitmproxy.ctx.master.server.config.certstore and + mitmproxy.ctx.master.server.config.certstore.default_ca + ) + if has_ca: + self.failed = mitmproxy.ctx.master.server.config.certstore.default_ca.has_expired() + if self.failed: + mitmproxy.ctx.master.add_log( + "The mitmproxy certificate authority has expired!\n" + "Please delete all CA-related files in your ~/.mitmproxy folder.\n" + "The CA will be regenerated automatically after restarting mitmproxy.\n" + "Then make sure all your clients have the new CA installed.", + "warn", + ) diff --git a/mitmproxy/addons/onboarding.py b/mitmproxy/addons/onboarding.py index 4b952438..cb57990f 100644 --- a/mitmproxy/addons/onboarding.py +++ b/mitmproxy/addons/onboarding.py @@ -8,9 +8,9 @@ class Onboarding(wsgiapp.WSGIApp): self.enabled = False def configure(self, options, updated): - self.host = options.app_host - self.port = options.app_port - self.enabled = options.app + self.host = options.onboarding_host + self.port = options.onboarding_port + self.enabled = options.onboarding def request(self, f): if self.enabled: diff --git a/mitmproxy/addons/replace.py b/mitmproxy/addons/replace.py index 09200d5d..34bb40c2 100644 --- a/mitmproxy/addons/replace.py +++ b/mitmproxy/addons/replace.py @@ -2,9 +2,47 @@ import re from mitmproxy import exceptions from mitmproxy import flowfilter +from mitmproxy import ctx -class Replace: +def parse_hook(s): + """ + Returns a (pattern, regex, replacement) tuple. + + The general form for a replacement hook is as follows: + + /patt/regex/replacement + + The first character specifies the separator. Example: + + :~q:foo:bar + + If only two clauses are specified, the pattern is set to match + universally (i.e. ".*"). Example: + + /foo/bar/ + + Clauses are parsed from left to right. Extra separators are taken to be + part of the final clause. For instance, the replacement clause below is + "foo/bar/": + + /one/two/foo/bar/ + """ + sep, rem = s[0], s[1:] + parts = rem.split(sep, 2) + if len(parts) == 2: + patt = ".*" + a, b = parts + elif len(parts) == 3: + patt, a, b = parts + else: + raise exceptions.OptionsError( + "Invalid replacement specifier: %s" % s + ) + return patt, a, b + + +class _ReplaceBase: def __init__(self): self.lst = [] @@ -16,9 +54,14 @@ class Replace: rex: a regular expression, as bytes. s: the replacement string, as bytes """ - if "replacements" in updated: + if self.optionName in updated: lst = [] - for fpatt, rex, s in options.replacements: + for rep in getattr(options, self.optionName): + if isinstance(rep, str): + fpatt, rex, s = parse_hook(rep) + else: + fpatt, rex, s = rep + flt = flowfilter.parse(fpatt) if not flt: raise exceptions.OptionsError( @@ -37,9 +80,9 @@ class Replace: for rex, s, flt in self.lst: if flt(f): if f.response: - f.response.replace(rex, s, flags=re.DOTALL) + self.replace(f.response, rex, s) else: - f.request.replace(rex, s, flags=re.DOTALL) + self.replace(f.request, rex, s) def request(self, flow): if not flow.reply.has_message: @@ -48,3 +91,22 @@ class Replace: def response(self, flow): if not flow.reply.has_message: self.execute(flow) + + +class Replace(_ReplaceBase): + optionName = "replacements" + + def replace(self, obj, rex, s): + obj.replace(rex, s, flags=re.DOTALL) + + +class ReplaceFile(_ReplaceBase): + optionName = "replacement_files" + + def replace(self, obj, rex, s): + try: + v = open(s, "rb").read() + except IOError as e: + ctx.log.warn("Could not read replacement file: %s" % s) + return + obj.replace(rex, v, flags=re.DOTALL) diff --git a/mitmproxy/addons/setheaders.py b/mitmproxy/addons/setheaders.py index 601e7521..95cf9a09 100644 --- a/mitmproxy/addons/setheaders.py +++ b/mitmproxy/addons/setheaders.py @@ -2,6 +2,43 @@ from mitmproxy import exceptions from mitmproxy import flowfilter +def parse_setheader(s): + """ + Returns a (pattern, regex, replacement) tuple. + + The general form for a replacement hook is as follows: + + /patt/regex/replacement + + The first character specifies the separator. Example: + + :~q:foo:bar + + If only two clauses are specified, the pattern is set to match + universally (i.e. ".*"). Example: + + /foo/bar/ + + Clauses are parsed from left to right. Extra separators are taken to be + part of the final clause. For instance, the replacement clause below is + "foo/bar/": + + /one/two/foo/bar/ + """ + sep, rem = s[0], s[1:] + parts = rem.split(sep, 2) + if len(parts) == 2: + patt = ".*" + a, b = parts + elif len(parts) == 3: + patt, a, b = parts + else: + raise exceptions.OptionsError( + "Invalid replacement specifier: %s" % s + ) + return patt, a, b + + class SetHeaders: def __init__(self): self.lst = [] @@ -16,7 +53,12 @@ class SetHeaders: """ if "setheaders" in updated: self.lst = [] - for fpatt, header, value in options.setheaders: + for shead in options.setheaders: + if isinstance(shead, str): + fpatt, header, value = parse_setheader(shead) + else: + fpatt, header, value = shead + flt = flowfilter.parse(fpatt) if not flt: raise exceptions.OptionsError( diff --git a/mitmproxy/addons/termlog.py b/mitmproxy/addons/termlog.py index b75f5f5a..5fdb6245 100644 --- a/mitmproxy/addons/termlog.py +++ b/mitmproxy/addons/termlog.py @@ -5,7 +5,7 @@ from mitmproxy import log class TermLog: - def __init__(self, outfile=sys.stdout): + def __init__(self, outfile=None): self.options = None self.outfile = outfile @@ -13,10 +13,15 @@ class TermLog: self.options = options def log(self, e): + if log.log_tier(e.level) == log.log_tier("error"): + outfile = self.outfile or sys.stderr + else: + outfile = self.outfile or sys.stdout + if self.options.verbosity >= log.log_tier(e.level): click.secho( e.msg, - file=self.outfile, + file=outfile, fg=dict(error="red", warn="yellow").get(e.level), dim=(e.level == "debug"), err=(e.level == "error") diff --git a/mitmproxy/addons/view.py b/mitmproxy/addons/view.py index 25696e43..8358a045 100644 --- a/mitmproxy/addons/view.py +++ b/mitmproxy/addons/view.py @@ -298,19 +298,19 @@ class View(collections.Sequence): "Invalid interception filter: %s" % opts.filter ) self.set_filter(filt) - if "order" in updated: - if opts.order is None: + if "console_order" in updated: + if opts.console_order is None: self.set_order(self.default_order) else: - if opts.order not in self.orders: + if opts.console_order not in self.orders: raise exceptions.OptionsError( - "Unknown flow order: %s" % opts.order + "Unknown flow order: %s" % opts.console_order ) - self.set_order(self.orders[opts.order]) - if "order_reversed" in updated: - self.set_reversed(opts.order_reversed) - if "focus_follow" in updated: - self.focus_follow = opts.focus_follow + self.set_order(self.orders[opts.console_order]) + if "console_order_reversed" in updated: + self.set_reversed(opts.console_order_reversed) + if "console_focus_follow" in updated: + self.focus_follow = opts.console_focus_follow def request(self, f): self.add(f) diff --git a/mitmproxy/certs.py b/mitmproxy/certs.py index 4e4eb4d1..4b939c80 100644 --- a/mitmproxy/certs.py +++ b/mitmproxy/certs.py @@ -3,8 +3,8 @@ import ssl import time import datetime import ipaddress - import sys + from pyasn1.type import univ, constraint, char, namedtype, tag from pyasn1.codec.der.decoder import decode from pyasn1.error import PyAsn1Error @@ -13,8 +13,8 @@ import OpenSSL from mitmproxy.types import serializable # Default expiry must not be too long: https://github.com/mitmproxy/mitmproxy/issues/815 - DEFAULT_EXP = 94608000 # = 24 * 60 * 60 * 365 * 3 + # Generated with "openssl dhparam". It's too slow to generate this on startup. DEFAULT_DHPARAM = b""" -----BEGIN DH PARAMETERS----- diff --git a/mitmproxy/options.py b/mitmproxy/options.py index 157b0168..630d2964 100644 --- a/mitmproxy/options.py +++ b/mitmproxy/options.py @@ -1,4 +1,4 @@ -from typing import Tuple, Optional, Sequence +from typing import Tuple, Optional, Sequence, Union from mitmproxy import optmanager @@ -22,96 +22,97 @@ DEFAULT_CLIENT_CIPHERS = "ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA class Options(optmanager.OptManager): def __init__( - self, - *, # all args are keyword-only. - # TODO: rename to onboarding_app_* - app: bool = True, - app_host: str = APP_HOST, - app_port: int = APP_PORT, - anticache: bool = False, - anticomp: bool = False, - client_replay: Sequence[str] = [], - replay_kill_extra: bool = False, - keepserving: bool = True, - no_server: bool = False, - server_replay_nopop: bool = False, - refresh_server_playback: bool = True, - rfile: Optional[str] = None, - scripts: Sequence[str] = [], - showhost: bool = False, - replacements: Sequence[Tuple[str, str, str]] = [], - server_replay_use_headers: Sequence[str] = [], - setheaders: Sequence[Tuple[str, str, str]] = [], - server_replay: Sequence[str] = [], - stickycookie: Optional[str] = None, - stickyauth: Optional[str] = None, - stream_large_bodies: Optional[int] = None, - verbosity: int = 2, - default_contentview: str = "auto", - streamfile: Optional[str] = None, - streamfile_append: bool = False, - server_replay_ignore_content: bool = False, - server_replay_ignore_params: Sequence[str] = [], - server_replay_ignore_payload_params: Sequence[str] = [], - server_replay_ignore_host: bool = False, - # Proxy options - auth_nonanonymous: bool = False, - auth_singleuser: Optional[str] = None, - auth_htpasswd: Optional[str] = None, - add_upstream_certs_to_client_chain: bool = False, - body_size_limit: Optional[int] = None, - cadir: str = CA_DIR, - certs: Sequence[Tuple[str, str]] = [], - ciphers_client: str=DEFAULT_CLIENT_CIPHERS, - ciphers_server: Optional[str]=None, - clientcerts: Optional[str] = None, - http2: bool = True, - ignore_hosts: Sequence[str] = [], - listen_host: str = "", - listen_port: int = LISTEN_PORT, - upstream_bind_address: str = "", - mode: str = "regular", - no_upstream_cert: bool = False, - rawtcp: bool = False, - websocket: bool = True, - spoof_source_address: bool = False, - upstream_server: Optional[str] = None, - upstream_auth: Optional[str] = None, - ssl_version_client: str = "secure", - ssl_version_server: str = "secure", - ssl_insecure: bool = False, - ssl_verify_upstream_trusted_cadir: Optional[str] = None, - ssl_verify_upstream_trusted_ca: Optional[str] = None, - tcp_hosts: Sequence[str] = [], - - intercept: Optional[str] = None, - - # Console options - eventlog: bool = False, - focus_follow: bool = False, - filter: Optional[str] = None, - palette: Optional[str] = "dark", - palette_transparent: bool = False, - no_mouse: bool = False, - order: Optional[str] = None, - order_reversed: bool = False, - - # Web options - open_browser: bool = True, - wdebug: bool = False, - wport: int = 8081, - wiface: str = "127.0.0.1", - - # Dump options - filtstr: Optional[str] = None, - flow_detail: int = 1 + self, + *, # all args are keyword-only. + onboarding: bool = True, + onboarding_host: str = APP_HOST, + onboarding_port: int = APP_PORT, + anticache: bool = False, + anticomp: bool = False, + client_replay: Sequence[str] = [], + replay_kill_extra: bool = False, + keepserving: bool = True, + no_server: bool = False, + server_replay_nopop: bool = False, + refresh_server_playback: bool = True, + rfile: Optional[str] = None, + scripts: Sequence[str] = [], + showhost: bool = False, + replacements: Sequence[Union[Tuple[str, str, str], str]] = [], + replacement_files: Sequence[Union[Tuple[str, str, str], str]] = [], + server_replay_use_headers: Sequence[str] = [], + setheaders: Sequence[Union[Tuple[str, str, str], str]] = [], + server_replay: Sequence[str] = [], + stickycookie: Optional[str] = None, + stickyauth: Optional[str] = None, + stream_large_bodies: Optional[int] = None, + verbosity: int = 2, + default_contentview: str = "auto", + streamfile: Optional[str] = None, + streamfile_append: bool = False, + server_replay_ignore_content: bool = False, + server_replay_ignore_params: Sequence[str] = [], + server_replay_ignore_payload_params: Sequence[str] = [], + server_replay_ignore_host: bool = False, + # Proxy options + auth_nonanonymous: bool = False, + auth_singleuser: Optional[str] = None, + auth_htpasswd: Optional[str] = None, + add_upstream_certs_to_client_chain: bool = False, + body_size_limit: Optional[int] = None, + cadir: str = CA_DIR, + certs: Sequence[Tuple[str, str]] = [], + ciphers_client: str=DEFAULT_CLIENT_CIPHERS, + ciphers_server: Optional[str]=None, + clientcerts: Optional[str] = None, + http2: bool = True, + ignore_hosts: Sequence[str] = [], + listen_host: str = "", + listen_port: int = LISTEN_PORT, + upstream_bind_address: str = "", + mode: str = "regular", + no_upstream_cert: bool = False, + rawtcp: bool = False, + websocket: bool = True, + spoof_source_address: bool = False, + upstream_server: Optional[str] = None, + upstream_auth: Optional[str] = None, + ssl_version_client: str = "secure", + ssl_version_server: str = "secure", + ssl_insecure: bool = False, + ssl_verify_upstream_trusted_cadir: Optional[str] = None, + ssl_verify_upstream_trusted_ca: Optional[str] = None, + tcp_hosts: Sequence[str] = [], + + intercept: Optional[str] = None, + + # Console options + console_eventlog: bool = False, + console_focus_follow: bool = False, + console_palette: Optional[str] = "dark", + console_palette_transparent: bool = False, + console_no_mouse: bool = False, + console_order: Optional[str] = None, + console_order_reversed: bool = False, + + filter: Optional[str] = None, + + # Web options + web_open_browser: bool = True, + web_debug: bool = False, + web_port: int = 8081, + web_iface: str = "127.0.0.1", + + # Dump options + filtstr: Optional[str] = None, + flow_detail: int = 1 ) -> None: # We could replace all assignments with clever metaprogramming, # but type hints are a much more valueable asset. - self.app = app - self.app_host = app_host - self.app_port = app_port + self.onboarding = onboarding + self.onboarding_host = onboarding_host + self.onboarding_port = onboarding_port self.anticache = anticache self.anticomp = anticomp self.client_replay = client_replay @@ -124,6 +125,7 @@ class Options(optmanager.OptManager): self.scripts = scripts self.showhost = showhost self.replacements = replacements + self.replacement_files = replacement_files self.server_replay_use_headers = server_replay_use_headers self.setheaders = setheaders self.server_replay = server_replay @@ -172,20 +174,21 @@ class Options(optmanager.OptManager): self.intercept = intercept # Console options - self.eventlog = eventlog - self.focus_follow = focus_follow + self.console_eventlog = console_eventlog + self.console_focus_follow = console_focus_follow + self.console_palette = console_palette + self.console_palette_transparent = console_palette_transparent + self.console_no_mouse = console_no_mouse + self.console_order = console_order + self.console_order_reversed = console_order_reversed + self.filter = filter - self.palette = palette - self.palette_transparent = palette_transparent - self.no_mouse = no_mouse - self.order = order - self.order_reversed = order_reversed # Web options - self.open_browser = open_browser - self.wdebug = wdebug - self.wport = wport - self.wiface = wiface + self.web_open_browser = web_open_browser + self.web_debug = web_debug + self.web_port = web_port + self.web_iface = web_iface # Dump options self.filtstr = filtstr diff --git a/mitmproxy/optmanager.py b/mitmproxy/optmanager.py index da1a3f17..f95ce836 100644 --- a/mitmproxy/optmanager.py +++ b/mitmproxy/optmanager.py @@ -229,7 +229,10 @@ class OptManager(metaclass=_DefaultsMeta): this object. May raise OptionsError if the config file is invalid. """ data = self._load(text) - self.update(**data) + try: + self.update(**data) + except KeyError as v: + raise exceptions.OptionsError(v) def load_paths(self, *paths): """ @@ -242,7 +245,12 @@ class OptManager(metaclass=_DefaultsMeta): if os.path.exists(p) and os.path.isfile(p): with open(p, "r") as f: txt = f.read() - self.load(txt) + try: + self.load(txt) + except exceptions.OptionsError as e: + raise exceptions.OptionsError( + "Error reading %s: %s" % (p, e) + ) def merge(self, opts): """ diff --git a/mitmproxy/test/tflow.py b/mitmproxy/test/tflow.py index 959c9a2c..edf4d7a7 100644 --- a/mitmproxy/test/tflow.py +++ b/mitmproxy/test/tflow.py @@ -1,9 +1,11 @@ from mitmproxy.test import tutils from mitmproxy import tcp +from mitmproxy import websocket from mitmproxy import controller from mitmproxy import http from mitmproxy import connections from mitmproxy import flow +from mitmproxy.net import http as net_http def ttcpflow(client_conn=True, server_conn=True, messages=True, err=None): @@ -26,6 +28,60 @@ def ttcpflow(client_conn=True, server_conn=True, messages=True, err=None): return f +def twebsocketflow(client_conn=True, server_conn=True, messages=True, err=None, handshake_flow=True): + + if client_conn is True: + client_conn = tclient_conn() + if server_conn is True: + server_conn = tserver_conn() + if handshake_flow is True: + req = http.HTTPRequest( + "relative", + "GET", + "http", + "example.com", + "80", + "/ws", + "HTTP/1.1", + headers=net_http.Headers( + connection="upgrade", + upgrade="websocket", + sec_websocket_version="13", + sec_websocket_key="1234", + ), + content=b'' + ) + resp = http.HTTPResponse( + "HTTP/1.1", + 101, + reason=net_http.status_codes.RESPONSES.get(101), + headers=net_http.Headers( + connection='upgrade', + upgrade='websocket', + sec_websocket_accept=b'', + ), + content=b'', + ) + handshake_flow = http.HTTPFlow(client_conn, server_conn) + handshake_flow.request = req + handshake_flow.response = resp + + f = websocket.WebSocketFlow(client_conn, server_conn, handshake_flow) + + if messages is True: + messages = [ + websocket.WebSocketBinaryMessage(f, True, b"hello binary"), + websocket.WebSocketTextMessage(f, False, "hello text".encode()), + ] + if err is True: + err = terr() + + f.messages = messages + f.error = err + f.reply = controller.DummyReply() + return f + + def tflow(client_conn=True, server_conn=True, req=True, resp=None, err=None): """ @type client_conn: bool | None | mitmproxy.proxy.connection.ClientConnection diff --git a/mitmproxy/tools/cmdline.py b/mitmproxy/tools/cmdline.py index f607ad71..1c620fd6 100644 --- a/mitmproxy/tools/cmdline.py +++ b/mitmproxy/tools/cmdline.py @@ -1,9 +1,7 @@ import argparse -import re import os from mitmproxy import exceptions -from mitmproxy import flowfilter from mitmproxy import options from mitmproxy import platform from mitmproxy.utils import human @@ -19,91 +17,6 @@ class ParseException(Exception): pass -def _parse_hook(s): - sep, rem = s[0], s[1:] - parts = rem.split(sep, 2) - if len(parts) == 2: - patt = ".*" - a, b = parts - elif len(parts) == 3: - patt, a, b = parts - else: - raise ParseException( - "Malformed hook specifier - too few clauses: %s" % s - ) - - if not a: - raise ParseException("Empty clause: %s" % str(patt)) - - if not flowfilter.parse(patt): - raise ParseException("Malformed filter pattern: %s" % patt) - - return patt, a, b - - -def parse_replace_hook(s): - """ - Returns a (pattern, regex, replacement) tuple. - - The general form for a replacement hook is as follows: - - /patt/regex/replacement - - The first character specifies the separator. Example: - - :~q:foo:bar - - If only two clauses are specified, the pattern is set to match - universally (i.e. ".*"). Example: - - /foo/bar/ - - Clauses are parsed from left to right. Extra separators are taken to be - part of the final clause. For instance, the replacement clause below is - "foo/bar/": - - /one/two/foo/bar/ - - Checks that pattern and regex are both well-formed. Raises - ParseException on error. - """ - patt, regex, replacement = _parse_hook(s) - try: - re.compile(regex) - except re.error as e: - raise ParseException("Malformed replacement regex: %s" % str(e)) - return patt, regex, replacement - - -def parse_setheader(s): - """ - Returns a (pattern, header, value) tuple. - - The general form for a replacement hook is as follows: - - /patt/header/value - - The first character specifies the separator. Example: - - :~q:foo:bar - - If only two clauses are specified, the pattern is set to match - universally (i.e. ".*"). Example: - - /foo/bar/ - - Clauses are parsed from left to right. Extra separators are taken to be - part of the final clause. For instance, the value clause below is - "foo/bar/": - - /one/two/foo/bar/ - - Checks that pattern and regex are both well-formed. Raises - ParseException on error. - """ - return _parse_hook(s) - - def get_common_options(args): stickycookie, stickyauth = None, None if args.stickycookie_filt: @@ -116,34 +29,6 @@ def get_common_options(args): if stream_large_bodies: stream_large_bodies = human.parse_size(stream_large_bodies) - reps = [] - for i in args.replace or []: - try: - p = parse_replace_hook(i) - except ParseException as e: - raise exceptions.OptionsError(e) - reps.append(p) - for i in args.replace_file or []: - try: - patt, rex, path = parse_replace_hook(i) - except ParseException as e: - raise exceptions.OptionsError(e) - try: - v = open(path, "rb").read() - except IOError as e: - raise exceptions.OptionsError( - "Could not read replace file: %s" % path - ) - reps.append((patt, rex, v)) - - setheaders = [] - for i in args.setheader or []: - try: - p = parse_setheader(i) - except ParseException as e: - raise exceptions.OptionsError(e) - setheaders.append(p) - if args.streamfile and args.streamfile[0] == args.rfile: if args.streamfile[1] == "wb": raise exceptions.OptionsError( @@ -212,9 +97,9 @@ def get_common_options(args): args.verbose = 0 return dict( - app=args.app, - app_host=args.app_host, - app_port=args.app_port, + onboarding=args.onboarding, + onboarding_host=args.onboarding_host, + onboarding_port=args.onboarding_port, anticache=args.anticache, anticomp=args.anticomp, @@ -224,8 +109,9 @@ def get_common_options(args): refresh_server_playback=not args.norefresh, server_replay_use_headers=args.server_replay_use_headers, rfile=args.rfile, - replacements=reps, - setheaders=setheaders, + replacements=args.replacements, + replacement_files=args.replacement_files, + setheaders=args.setheaders, server_replay=args.server_replay, scripts=args.scripts, stickycookie=stickycookie, @@ -568,13 +454,13 @@ def proxy_ssl_options(parser): def onboarding_app(parser): group = parser.add_argument_group("Onboarding App") group.add_argument( - "--noapp", - action="store_false", dest="app", + "--no-onboarding", + action="store_false", dest="onboarding", help="Disable the mitmproxy onboarding app." ) group.add_argument( - "--app-host", - action="store", dest="app_host", + "--onboarding-host", + action="store", dest="onboarding_host", help=""" Domain to serve the onboarding app from. For transparent mode, use an IP when a DNS entry for the app domain is not present. Default: @@ -582,9 +468,9 @@ def onboarding_app(parser): """ % options.APP_HOST ) group.add_argument( - "--app-port", + "--onboarding-port", action="store", - dest="app_port", + dest="onboarding_port", type=int, metavar="80", help="Port to serve the onboarding app from." @@ -676,13 +562,13 @@ def replacements(parser): ) group.add_argument( "--replace", - action="append", type=str, dest="replace", + action="append", type=str, dest="replacements", metavar="PATTERN", help="Replacement pattern." ) group.add_argument( "--replace-from-file", - action="append", type=str, dest="replace_file", + action="append", type=str, dest="replacement_files", metavar="PATH", help=""" Replacement pattern, where the replacement clause is a path to a @@ -702,7 +588,7 @@ def set_headers(parser): ) group.add_argument( "--setheader", - action="append", type=str, dest="setheader", + action="append", type=str, dest="setheaders", metavar="PATTERN", help="Header set pattern." ) @@ -762,42 +648,42 @@ def common_options(parser): def mitmproxy(): - # Don't import mitmproxy.tools.console for mitmdump, urwid is not available on all - # platforms. + # Don't import mitmproxy.tools.console for mitmdump, urwid is not available + # on all platforms. from .console import palettes parser = argparse.ArgumentParser(usage="%(prog)s [options]") common_options(parser) parser.add_argument( "--palette", type=str, - action="store", dest="palette", + action="store", dest="console_palette", choices=sorted(palettes.palettes.keys()), help="Select color palette: " + ", ".join(palettes.palettes.keys()) ) parser.add_argument( "--palette-transparent", - action="store_true", dest="palette_transparent", + action="store_true", dest="console_palette_transparent", help="Set transparent background for palette." ) parser.add_argument( "-e", "--eventlog", - action="store_true", dest="eventlog", + action="store_true", dest="console_eventlog", help="Show event log." ) parser.add_argument( "--follow", - action="store_true", dest="focus_follow", + action="store_true", dest="console_focus_follow", help="Focus follows new flows." ) parser.add_argument( "--order", - type=str, dest="order", + type=str, dest="console_order", choices=[o[1] for o in view.orders], help="Flow sort order." ) parser.add_argument( "--no-mouse", - action="store_true", dest="no_mouse", + action="store_true", dest="console_no_mouse", help="Disable mouse interaction." ) group = parser.add_argument_group( @@ -851,24 +737,24 @@ def mitmweb(): group = parser.add_argument_group("Mitmweb") group.add_argument( "--no-browser", - action="store_false", dest="open_browser", + action="store_false", dest="web_open_browser", help="Don't start a browser" ) group.add_argument( - "--wport", - action="store", type=int, dest="wport", + "--web-port", + action="store", type=int, dest="web_port", metavar="PORT", help="Mitmweb port." ) group.add_argument( - "--wiface", - action="store", dest="wiface", + "--web-iface", + action="store", dest="web_iface", metavar="IFACE", help="Mitmweb interface." ) group.add_argument( - "--wdebug", - action="store_true", dest="wdebug", + "--web-debug", + action="store_true", dest="web_debug", help="Turn on mitmweb debugging" ) diff --git a/mitmproxy/tools/console/flowlist.py b/mitmproxy/tools/console/flowlist.py index 1fe0be73..8ee2d55d 100644 --- a/mitmproxy/tools/console/flowlist.py +++ b/mitmproxy/tools/console/flowlist.py @@ -387,7 +387,7 @@ class FlowListBox(urwid.ListBox): lookup = dict([(i[0], i[1]) for i in view.orders]) def change_order(k): - self.master.options.order = lookup[k] + self.master.options.console_order = lookup[k] signals.status_prompt_onekey.send( prompt = "Order", @@ -398,8 +398,8 @@ class FlowListBox(urwid.ListBox): o = self.master.options o.focus_follow = not o.focus_follow elif key == "v": - val = not self.master.options.order_reversed - self.master.options.order_reversed = val + val = not self.master.options.console_order_reversed + self.master.options.console_order_reversed = val elif key == "W": if self.master.options.streamfile: self.master.options.streamfile = None diff --git a/mitmproxy/tools/console/master.py b/mitmproxy/tools/console/master.py index 8afdce2c..77b48e8d 100644 --- a/mitmproxy/tools/console/master.py +++ b/mitmproxy/tools/console/master.py @@ -33,8 +33,6 @@ from mitmproxy.tools.console import statusbar from mitmproxy.tools.console import window from mitmproxy.utils import strutils -from mitmproxy.net import tcp - EVENTLOG_SIZE = 10000 @@ -108,7 +106,7 @@ class ConsoleMaster(master.Master): self.logbuffer.append(e) if len(self.logbuffer) > EVENTLOG_SIZE: self.logbuffer.pop(0) - if self.options.focus_follow: + if self.options.console_focus_follow: self.logbuffer.set_focus(len(self.logbuffer) - 1) def sig_call_in(self, sender, seconds, callback, args=()): @@ -152,7 +150,7 @@ class ConsoleMaster(master.Master): signals.add_log("Script error: %s" % e, "warn") def toggle_eventlog(self): - self.options.eventlog = not self.options.eventlog + self.options.console_eventlog = not self.options.console_eventlog self.view_flowlist() signals.replace_view_state.send(self) @@ -232,8 +230,8 @@ class ConsoleMaster(master.Master): def set_palette(self, options, updated): self.ui.register_palette( - palettes.palettes[options.palette].palette( - options.palette_transparent + palettes.palettes[options.console_palette].palette( + options.console_palette_transparent ) ) self.ui.clear() @@ -255,7 +253,7 @@ class ConsoleMaster(master.Master): self.loop = urwid.MainLoop( urwid.SolidFill("x"), screen = self.ui, - handle_mouse = not self.options.no_mouse, + handle_mouse = not self.options.console_no_mouse, ) self.ab = statusbar.ActionBar() @@ -273,14 +271,6 @@ class ConsoleMaster(master.Master): sys.exit(1) self.loop.set_alarm_in(0.01, self.ticker) - if self.options.http2 and not tcp.HAS_ALPN: # pragma: no cover - def http2err(*args, **kwargs): - signals.status_message.send( - message = "HTTP/2 disabled - OpenSSL 1.0.2+ required." - " Use --no-http2 to silence this warning.", - expire=5 - ) - self.loop.set_alarm_in(0.01, http2err) self.loop.set_alarm_in( 0.0001, @@ -357,7 +347,7 @@ class ConsoleMaster(master.Master): if self.ui.started: self.ui.clear() - if self.options.eventlog: + if self.options.console_eventlog: body = flowlist.BodyPile(self) else: body = flowlist.FlowListBox(self) diff --git a/mitmproxy/tools/console/options.py b/mitmproxy/tools/console/options.py index 54876f87..4115bd18 100644 --- a/mitmproxy/tools/console/options.py +++ b/mitmproxy/tools/console/options.py @@ -6,6 +6,9 @@ from mitmproxy.tools.console import grideditor from mitmproxy.tools.console import select from mitmproxy.tools.console import signals +from mitmproxy.addons import replace +from mitmproxy.addons import setheaders + footer = [ ('heading_key', "enter/space"), ":toggle ", ('heading_key', "C"), ":clear all ", @@ -75,7 +78,7 @@ class Options(urwid.WidgetWrap): select.Option( "Palette", "P", - checker("palette", master.options), + checker("console_palette", master.options), self.palette ), select.Option( @@ -188,10 +191,16 @@ class Options(urwid.WidgetWrap): ) def setheaders(self): + data = [] + for d in self.master.options.setheaders: + if isinstance(d, str): + data.append(setheaders.parse_setheader(d)) + else: + data.append(d) self.master.view_grideditor( grideditor.SetHeadersEditor( self.master, - self.master.options.setheaders, + data, self.master.options.setter("setheaders") ) ) @@ -215,10 +224,16 @@ class Options(urwid.WidgetWrap): ) def replacepatterns(self): + data = [] + for d in self.master.options.replacements: + if isinstance(d, str): + data.append(replace.parse_hook(d)) + else: + data.append(d) self.master.view_grideditor( grideditor.ReplaceEditor( self.master, - self.master.options.replacements, + data, self.master.options.setter("replacements") ) ) diff --git a/mitmproxy/tools/console/palettepicker.py b/mitmproxy/tools/console/palettepicker.py index 0d943baf..4c5c62a0 100644 --- a/mitmproxy/tools/console/palettepicker.py +++ b/mitmproxy/tools/console/palettepicker.py @@ -42,7 +42,7 @@ class PalettePicker(urwid.WidgetWrap): return select.Option( i, None, - lambda: self.master.options.palette == name, + lambda: self.master.options.console_palette == name, lambda: setattr(self.master.options, "palette", name) ) @@ -58,7 +58,7 @@ class PalettePicker(urwid.WidgetWrap): select.Option( "Transparent", "T", - lambda: master.options.palette_transparent, + lambda: master.options.console_palette_transparent, master.options.toggler("palette_transparent") ) ] diff --git a/mitmproxy/tools/console/statusbar.py b/mitmproxy/tools/console/statusbar.py index 785dc766..2c7f9efb 100644 --- a/mitmproxy/tools/console/statusbar.py +++ b/mitmproxy/tools/console/statusbar.py @@ -204,10 +204,10 @@ class StatusBar(urwid.WidgetWrap): r.append("[") r.append(("heading_key", "M")) r.append(":%s]" % self.master.options.default_contentview) - if self.master.options.order: + if self.master.options.console_order: r.append("[") r.append(("heading_key", "o")) - r.append(":%s]" % self.master.options.order) + r.append(":%s]" % self.master.options.console_order) opts = [] if self.master.options.anticache: @@ -222,7 +222,7 @@ class StatusBar(urwid.WidgetWrap): opts.append("killextra") if self.master.options.no_upstream_cert: opts.append("no-upstream-cert") - if self.master.options.focus_follow: + if self.master.options.console_focus_follow: opts.append("following") if self.master.options.stream_large_bodies: opts.append( @@ -258,7 +258,7 @@ class StatusBar(urwid.WidgetWrap): else: offset = self.master.view.focus.index + 1 - if self.master.options.order_reversed: + if self.master.options.console_order_reversed: arrow = common.SYMBOL_UP else: arrow = common.SYMBOL_DOWN diff --git a/mitmproxy/tools/dump.py b/mitmproxy/tools/dump.py index 4e2844a1..e1e40fb0 100644 --- a/mitmproxy/tools/dump.py +++ b/mitmproxy/tools/dump.py @@ -6,7 +6,6 @@ from mitmproxy import addons from mitmproxy import options from mitmproxy import master from mitmproxy.addons import dumper, termlog -from mitmproxy.net import tcp class DumpError(Exception): @@ -30,7 +29,13 @@ class Options(options.Options): class DumpMaster(master.Master): - def __init__(self, options, server, with_termlog=True, with_dumper=True): + def __init__( + self, + options: Options, + server, + with_termlog=True, + with_dumper=True, + ) -> None: master.Master.__init__(self, options, server) self.has_errored = False if with_termlog: @@ -38,8 +43,6 @@ class DumpMaster(master.Master): self.addons.add(*addons.default_addons()) if with_dumper: self.addons.add(dumper.Dumper()) - # This line is just for type hinting - self.options = self.options # type: Options if not self.options.no_server: self.add_log( @@ -47,13 +50,6 @@ class DumpMaster(master.Master): "info" ) - if self.server and self.options.http2 and not tcp.HAS_ALPN: # pragma: no cover - self.add_log( - "ALPN support missing (OpenSSL 1.0.2+ required)!\n" - "HTTP/2 is disabled. Use --no-http2 to silence this warning.", - "error" - ) - if options.rfile: try: self.load_flows_file(options.rfile) diff --git a/mitmproxy/tools/main.py b/mitmproxy/tools/main.py index 727e993f..ce78cd13 100644 --- a/mitmproxy/tools/main.py +++ b/mitmproxy/tools/main.py @@ -70,14 +70,15 @@ def mitmproxy(args=None): # pragma: no cover console_options.merge(cmdline.get_common_options(args)) console_options.merge( dict( - palette = args.palette, - palette_transparent = args.palette_transparent, - eventlog = args.eventlog, - focus_follow = args.focus_follow, - intercept = args.intercept, + console_palette = args.console_palette, + console_palette_transparent = args.console_palette_transparent, + console_eventlog = args.console_eventlog, + console_focus_follow = args.console_focus_follow, + console_no_mouse = args.console_no_mouse, + console_order = args.console_order, + filter = args.filter, - no_mouse = args.no_mouse, - order = args.order, + intercept = args.intercept, ) ) @@ -149,10 +150,10 @@ def mitmweb(args=None): # pragma: no cover web_options.merge( dict( intercept = args.intercept, - open_browser = args.open_browser, - wdebug = args.wdebug, - wiface = args.wiface, - wport = args.wport, + web_open_browser = args.web_open_browser, + web_debug = args.web_debug, + web_iface = args.web_iface, + web_port = args.web_port, ) ) server = process_options(parser, web_options, args) diff --git a/mitmproxy/tools/web/master.py b/mitmproxy/tools/web/master.py index e6c15a0a..6ebcfe47 100644 --- a/mitmproxy/tools/web/master.py +++ b/mitmproxy/tools/web/master.py @@ -38,7 +38,7 @@ class WebMaster(master.Master): if with_termlog: self.addons.add(termlog.TermLog()) self.app = app.Application( - self, self.options.wdebug + self, self.options.web_debug ) # This line is just for type hinting self.options = self.options # type: Options @@ -103,7 +103,7 @@ class WebMaster(master.Master): iol = tornado.ioloop.IOLoop.instance() http_server = tornado.httpserver.HTTPServer(self.app) - http_server.listen(self.options.wport, self.options.wiface) + http_server.listen(self.options.web_port, self.options.web_iface) iol.add_callback(self.start) tornado.ioloop.PeriodicCallback(lambda: self.tick(timeout=0), 5).start() @@ -113,13 +113,13 @@ class WebMaster(master.Master): "info" ) - web_url = "http://{}:{}/".format(self.options.wiface, self.options.wport) + web_url = "http://{}:{}/".format(self.options.web_iface, self.options.web_port) self.add_log( "Web server listening at {}".format(web_url), "info" ) - if self.options.open_browser: + if self.options.web_open_browser: success = open_browser(web_url) if not success: self.add_log( diff --git a/mitmproxy/utils/debug.py b/mitmproxy/utils/debug.py index 93fefa9d..52f48f6b 100644 --- a/mitmproxy/utils/debug.py +++ b/mitmproxy/utils/debug.py @@ -19,7 +19,21 @@ def dump_system_info(): try: c = ['git', 'describe', '--tags', '--long'] git_describe = subprocess.check_output(c, stderr=subprocess.STDOUT) - git_describe = git_describe.decode().strip() + last_tag, tag_dist, commit = git_describe.decode().strip().rsplit("-", 2) + + if last_tag.startswith('v'): + # remove the 'v' prefix + last_tag = last_tag[1:] + if commit.startswith('g'): + # remove the 'g' prefix added by recent git versions + commit = commit[1:] + + # build the same version specifier as used for snapshots by rtool + git_describe = "{version}dev{tag:04}-0x{commit}".format( + version=last_tag, + tag=int(tag_dist), + commit=commit, + ) except: pass diff --git a/release/rtool.py b/release/rtool.py index 59899510..271392ba 100755 --- a/release/rtool.py +++ b/release/rtool.py @@ -89,6 +89,10 @@ def get_snapshot_version() -> str: if tag_dist == 0: return get_version() else: + # remove the 'g' prefix added by recent git versions + if commit.startswith('g'): + commit = commit[1:] + # The wheel build tag (we use the commit) must start with a digit, so we include "0x" return "{version}dev{tag_dist:04}-0x{commit}".format( version=get_version(), # this should already be the next version @@ -100,7 +100,6 @@ setup( "mypy-lang>=0.4.6, <0.5", "rstcheck>=2.2, <4.0", "tox>=2.3, <3", - "mock>=2.0, <2.1", "pytest>=3, <3.1", "pytest-cov>=2.2.1, <3", "pytest-timeout>=1.0.0, <2", diff --git a/test/mitmproxy/addons/test_check_alpn.py b/test/mitmproxy/addons/test_check_alpn.py new file mode 100644 index 00000000..2dc0c835 --- /dev/null +++ b/test/mitmproxy/addons/test_check_alpn.py @@ -0,0 +1,23 @@ +from mitmproxy.addons import check_alpn +from mitmproxy.test import taddons +from ...conftest import requires_alpn + + +class TestCheckALPN: + + @requires_alpn + def test_check_alpn(self): + msg = 'ALPN support missing' + + with taddons.context() as tctx: + a = check_alpn.CheckALPN() + tctx.configure(a) + assert not any(msg in m for l, m in tctx.master.event_log) + + def test_check_no_alpn(self, disable_alpn): + msg = 'ALPN support missing' + + with taddons.context() as tctx: + a = check_alpn.CheckALPN() + tctx.configure(a) + assert any(msg in m for l, m in tctx.master.event_log) diff --git a/test/mitmproxy/addons/test_check_ca.py b/test/mitmproxy/addons/test_check_ca.py new file mode 100644 index 00000000..fc64621c --- /dev/null +++ b/test/mitmproxy/addons/test_check_ca.py @@ -0,0 +1,19 @@ +import pytest +from unittest import mock + +from mitmproxy.addons import check_ca +from mitmproxy.test import taddons + + +class TestCheckCA: + + @pytest.mark.parametrize('expired', [False, True]) + def test_check_ca(self, expired): + msg = 'The mitmproxy certificate authority has expired!' + + with taddons.context() as tctx: + tctx.master.server = mock.MagicMock() + tctx.master.server.config.certstore.default_ca.has_expired = mock.MagicMock(return_value=expired) + a = check_ca.CheckCA() + tctx.configure(a) + assert any(msg in m for l, m in tctx.master.event_log) is expired diff --git a/test/mitmproxy/addons/test_clientplayback.py b/test/mitmproxy/addons/test_clientplayback.py index 32eca536..ef1f0782 100644 --- a/test/mitmproxy/addons/test_clientplayback.py +++ b/test/mitmproxy/addons/test_clientplayback.py @@ -1,5 +1,5 @@ import os -import mock +from unittest import mock from mitmproxy.test import tflow from mitmproxy.test import tutils @@ -33,7 +33,7 @@ class TestClientPlayback: with mock.patch(RP) as rp: assert not cp.current_thread cp.tick() - rp.assert_called() + assert rp.called assert cp.current_thread cp.keepserving = False @@ -41,7 +41,7 @@ class TestClientPlayback: cp.current_thread = None with mock.patch("mitmproxy.master.Master.shutdown") as sd: cp.tick() - sd.assert_called() + assert sd.called cp.current_thread = MockThread() with mock.patch("mitmproxy.master.Master.shutdown") as sd: diff --git a/test/mitmproxy/addons/test_disable_h2c_upgrade.py b/test/mitmproxy/addons/test_disable_h2c_upgrade.py new file mode 100644 index 00000000..6cab713d --- /dev/null +++ b/test/mitmproxy/addons/test_disable_h2c_upgrade.py @@ -0,0 +1,17 @@ +from mitmproxy.addons import disable_h2c_upgrade +from mitmproxy.test import tflow + + +class TestTermLog: + def test_simple(self): + a = disable_h2c_upgrade.DisableH2CleartextUpgrade() + + f = tflow.tflow() + f.request.headers['upgrade'] = 'h2c' + f.request.headers['connection'] = 'foo' + f.request.headers['http2-settings'] = 'bar' + + a.request(f) + assert 'upgrade' not in f.request.headers + assert 'connection' not in f.request.headers + assert 'http2-settings' not in f.request.headers diff --git a/test/mitmproxy/addons/test_dumper.py b/test/mitmproxy/addons/test_dumper.py index d8d2deeb..8fa8a22a 100644 --- a/test/mitmproxy/addons/test_dumper.py +++ b/test/mitmproxy/addons/test_dumper.py @@ -1,4 +1,7 @@ import io +import shutil +from unittest import mock + from mitmproxy.test import tflow from mitmproxy.test import taddons from mitmproxy.test import tutils @@ -7,8 +10,6 @@ from mitmproxy.addons import dumper from mitmproxy import exceptions from mitmproxy.tools import dump from mitmproxy import http -import shutil -import mock def test_configure(): @@ -156,7 +157,7 @@ def test_tcp(): d = dumper.Dumper(sio) with taddons.context(options=dump.Options()) as ctx: ctx.configure(d, flow_detail=3, showhost=True) - f = tflow.ttcpflow(client_conn=True, server_conn=True) + f = tflow.ttcpflow() d.tcp_message(f) assert "it's me" in sio.getvalue() sio.truncate(0) @@ -164,3 +165,21 @@ def test_tcp(): f = tflow.ttcpflow(client_conn=True, err=True) d.tcp_error(f) assert "Error in TCP" in sio.getvalue() + + +def test_websocket(): + sio = io.StringIO() + d = dumper.Dumper(sio) + with taddons.context(options=dump.Options()) as ctx: + ctx.configure(d, flow_detail=3, showhost=True) + f = tflow.twebsocketflow() + d.websocket_message(f) + assert "hello text" in sio.getvalue() + sio.truncate(0) + + d.websocket_end(f) + assert "WebSocket connection closed by" in sio.getvalue() + + f = tflow.twebsocketflow(client_conn=True, err=True) + d.websocket_error(f) + assert "Error in WebSocket" in sio.getvalue() diff --git a/test/mitmproxy/addons/test_evenstore.py b/test/mitmproxy/addons/test_evenstore.py index 78eb3287..f54b9980 100644 --- a/test/mitmproxy/addons/test_evenstore.py +++ b/test/mitmproxy/addons/test_evenstore.py @@ -1,4 +1,4 @@ -import mock +from unittest import mock from mitmproxy import log from mitmproxy.addons import eventstore diff --git a/test/mitmproxy/addons/test_replace.py b/test/mitmproxy/addons/test_replace.py index 34fa5017..ec38b77d 100644 --- a/test/mitmproxy/addons/test_replace.py +++ b/test/mitmproxy/addons/test_replace.py @@ -1,57 +1,59 @@ +import os.path from mitmproxy.test import tflow from mitmproxy.test import tutils from .. import tservers from mitmproxy.addons import replace -from mitmproxy import master -from mitmproxy import options -from mitmproxy import proxy +from mitmproxy.test import taddons class TestReplace: + def test_parse_hook(self): + x = replace.parse_hook("/foo/bar/voing") + assert x == ("foo", "bar", "voing") + x = replace.parse_hook("/foo/bar/vo/ing/") + assert x == ("foo", "bar", "vo/ing/") + x = replace.parse_hook("/bar/voing") + assert x == (".*", "bar", "voing") + tutils.raises("invalid replacement", replace.parse_hook, "/") + def test_configure(self): r = replace.Replace() - updated = set(["replacements"]) - r.configure(options.Options( - replacements=[("one", "two", "three")] - ), updated) - tutils.raises( - "invalid filter pattern", - r.configure, - options.Options( + with taddons.context() as tctx: + tctx.configure(r, replacements=[("one", "two", "three")]) + tutils.raises( + "invalid filter pattern", + tctx.configure, + r, replacements=[("~b", "two", "three")] - ), - updated - ) - tutils.raises( - "invalid regular expression", - r.configure, - options.Options( + ) + tutils.raises( + "invalid regular expression", + tctx.configure, + r, replacements=[("foo", "+", "three")] - ), - updated - ) + ) + tctx.configure(r, replacements=["/a/b/c/"]) def test_simple(self): - o = options.Options( - replacements = [ - ("~q", "foo", "bar"), - ("~s", "foo", "bar"), - ] - ) - m = master.Master(o, proxy.DummyServer()) - sa = replace.Replace() - m.addons.add(sa) - - f = tflow.tflow() - f.request.content = b"foo" - m.request(f) - assert f.request.content == b"bar" + r = replace.Replace() + with taddons.context() as tctx: + tctx.configure( + r, + replacements = [ + ("~q", "foo", "bar"), + ("~s", "foo", "bar"), + ] + ) + f = tflow.tflow() + f.request.content = b"foo" + r.request(f) + assert f.request.content == b"bar" - f = tflow.tflow(resp=True) - f.response.content = b"foo" - m.response(f) - assert f.response.content == b"bar" + f = tflow.tflow(resp=True) + f.response.content = b"foo" + r.response(f) + assert f.response.content == b"bar" class TestUpstreamProxy(tservers.HTTPUpstreamProxyTest): @@ -72,3 +74,36 @@ class TestUpstreamProxy(tservers.HTTPUpstreamProxyTest): req = p.request("get:'%s/p/418:b\"foo\"'" % self.server.urlbase) assert req.content == b"ORLY" assert req.status_code == 418 + + +class TestReplaceFile: + def test_simple(self): + r = replace.ReplaceFile() + with tutils.tmpdir() as td: + rp = os.path.join(td, "replacement") + with open(rp, "w") as f: + f.write("bar") + with taddons.context() as tctx: + tctx.configure( + r, + replacement_files = [ + ("~q", "foo", rp), + ("~s", "foo", rp), + ("~b nonexistent", "nonexistent", "nonexistent"), + ] + ) + f = tflow.tflow() + f.request.content = b"foo" + r.request(f) + assert f.request.content == b"bar" + + f = tflow.tflow(resp=True) + f.response.content = b"foo" + r.response(f) + assert f.response.content == b"bar" + + f = tflow.tflow() + f.request.content = b"nonexistent" + assert not tctx.master.event_log + r.request(f) + assert tctx.master.event_log diff --git a/test/mitmproxy/addons/test_serverplayback.py b/test/mitmproxy/addons/test_serverplayback.py index 613a290c..37766dfa 100644 --- a/test/mitmproxy/addons/test_serverplayback.py +++ b/test/mitmproxy/addons/test_serverplayback.py @@ -1,4 +1,6 @@ import os +import urllib + from mitmproxy.test import tutils from mitmproxy.test import tflow from mitmproxy.test import taddons @@ -22,8 +24,13 @@ def test_config(): with taddons.context() as tctx: fpath = os.path.join(p, "flows") tdump(fpath, [tflow.tflow(resp=True)]) - tctx.configure(s, server_replay = [fpath]) - tutils.raises(exceptions.OptionsError, tctx.configure, s, server_replay = [p]) + tctx.configure(s, server_replay=[fpath]) + tutils.raises( + exceptions.OptionsError, + tctx.configure, + s, + server_replay=[p] + ) def test_tick(): @@ -246,7 +253,7 @@ def test_ignore_params(): assert not s._hash(r) == s._hash(r2) -def test_ignore_payload_params(): +def thash(r, r2, setter): s = serverplayback.ServerPlayback() s.configure( options.Options( @@ -255,31 +262,59 @@ def test_ignore_payload_params(): [] ) - r = tflow.tflow(resp=True) - r.request.headers["Content-Type"] = "application/x-www-form-urlencoded" - r.request.content = b"paramx=x¶m1=1" - r2 = tflow.tflow(resp=True) - r2.request.headers["Content-Type"] = "application/x-www-form-urlencoded" - r2.request.content = b"paramx=x¶m1=1" + setter(r, paramx="x", param1="1") + + setter(r2, paramx="x", param1="1") # same parameters assert s._hash(r) == s._hash(r2) # ignored parameters != - r2.request.content = b"paramx=x¶m1=2" + setter(r2, paramx="x", param1="2") assert s._hash(r) == s._hash(r2) # missing parameter - r2.request.content = b"paramx=x" + setter(r2, paramx="x") assert s._hash(r) == s._hash(r2) # ignorable parameter added - r2.request.content = b"paramx=x¶m1=2" + setter(r2, paramx="x", param1="2") assert s._hash(r) == s._hash(r2) # not ignorable parameter changed - r2.request.content = b"paramx=y¶m1=1" + setter(r2, paramx="y", param1="1") assert not s._hash(r) == s._hash(r2) # not ignorable parameter missing + setter(r2, param1="1") r2.request.content = b"param1=1" assert not s._hash(r) == s._hash(r2) +def test_ignore_payload_params(): + def urlencode_setter(r, **kwargs): + r.request.content = urllib.parse.urlencode(kwargs).encode() + + r = tflow.tflow(resp=True) + r.request.headers["Content-Type"] = "application/x-www-form-urlencoded" + r2 = tflow.tflow(resp=True) + r2.request.headers["Content-Type"] = "application/x-www-form-urlencoded" + thash(r, r2, urlencode_setter) + + boundary = 'somefancyboundary' + + def multipart_setter(r, **kwargs): + b = "--{0}\n".format(boundary) + parts = [] + for k, v in kwargs.items(): + parts.append( + "Content-Disposition: form-data; name=\"%s\"\n\n" + "%s\n" % (k, v) + ) + c = b + b.join(parts) + b + r.request.content = c.encode() + r.request.headers["content-type"] = 'multipart/form-data; boundary=' +\ + boundary + + r = tflow.tflow(resp=True) + r2 = tflow.tflow(resp=True) + thash(r, r2, multipart_setter) + + def test_server_playback_full(): s = serverplayback.ServerPlayback() with taddons.context() as tctx: diff --git a/test/mitmproxy/addons/test_setheaders.py b/test/mitmproxy/addons/test_setheaders.py index 34395ddf..a721c180 100644 --- a/test/mitmproxy/addons/test_setheaders.py +++ b/test/mitmproxy/addons/test_setheaders.py @@ -3,19 +3,28 @@ from mitmproxy.test import tutils from mitmproxy.test import taddons from mitmproxy.addons import setheaders -from mitmproxy import options class TestSetHeaders: + def test_parse_setheaders(self): + x = setheaders.parse_setheader("/foo/bar/voing") + assert x == ("foo", "bar", "voing") + x = setheaders.parse_setheader("/foo/bar/vo/ing/") + assert x == ("foo", "bar", "vo/ing/") + x = setheaders.parse_setheader("/bar/voing") + assert x == (".*", "bar", "voing") + tutils.raises("invalid replacement", setheaders.parse_setheader, "/") + def test_configure(self): sh = setheaders.SetHeaders() - o = options.Options( - setheaders = [("~b", "one", "two")] - ) - tutils.raises( - "invalid setheader filter pattern", - sh.configure, o, o.keys() - ) + with taddons.context() as tctx: + tutils.raises( + "invalid setheader filter pattern", + tctx.configure, + sh, + setheaders = [("~b", "one", "two")] + ) + tctx.configure(sh, setheaders = ["/foo/bar/voing"]) def test_setheaders(self): sh = setheaders.SetHeaders() diff --git a/test/mitmproxy/addons/test_stickyauth.py b/test/mitmproxy/addons/test_stickyauth.py index df74f44d..a21ad38f 100644 --- a/test/mitmproxy/addons/test_stickyauth.py +++ b/test/mitmproxy/addons/test_stickyauth.py @@ -10,7 +10,15 @@ def test_configure(): r = stickyauth.StickyAuth() with taddons.context() as tctx: tctx.configure(r, stickyauth="~s") - tutils.raises(exceptions.OptionsError, tctx.configure, r, stickyauth="~~") + tutils.raises( + exceptions.OptionsError, + tctx.configure, + r, + stickyauth="~~" + ) + + tctx.configure(r, stickyauth=None) + assert not r.flt def test_simple(): diff --git a/test/mitmproxy/addons/test_stickycookie.py b/test/mitmproxy/addons/test_stickycookie.py index df5d0221..4a1ede8e 100644 --- a/test/mitmproxy/addons/test_stickycookie.py +++ b/test/mitmproxy/addons/test_stickycookie.py @@ -3,7 +3,6 @@ from mitmproxy.test import tutils from mitmproxy.test import taddons from mitmproxy.addons import stickycookie -from mitmproxy import options from mitmproxy.test import tutils as ntutils @@ -15,11 +14,15 @@ def test_domain_match(): class TestStickyCookie: def test_config(self): sc = stickycookie.StickyCookie() - o = options.Options(stickycookie = "~b") - tutils.raises( - "invalid filter", - sc.configure, o, o.keys() - ) + with taddons.context() as tctx: + tutils.raises( + "invalid filter", tctx.configure, sc, stickycookie="~b" + ) + + tctx.configure(sc, stickycookie="foo") + assert sc.flt + tctx.configure(sc, stickycookie=None) + assert not sc.flt def test_simple(self): sc = stickycookie.StickyCookie() diff --git a/test/mitmproxy/addons/test_streamfile.py b/test/mitmproxy/addons/test_streamfile.py index 82a4345b..4c60cd51 100644 --- a/test/mitmproxy/addons/test_streamfile.py +++ b/test/mitmproxy/addons/test_streamfile.py @@ -22,6 +22,10 @@ def test_configure(): "invalid filter", tctx.configure, sa, streamfile=p, filtstr="~~" ) + tctx.configure(sa, filtstr="foo") + assert sa.filt + tctx.configure(sa, filtstr=None) + assert not sa.filt def rd(p): diff --git a/test/mitmproxy/addons/test_termlog.py b/test/mitmproxy/addons/test_termlog.py index d9e18134..70c3a7f2 100644 --- a/test/mitmproxy/addons/test_termlog.py +++ b/test/mitmproxy/addons/test_termlog.py @@ -1,16 +1,27 @@ -import io +import sys +import pytest from mitmproxy.addons import termlog from mitmproxy import log -from mitmproxy.tools import dump +from mitmproxy.tools.dump import Options +from mitmproxy.test import taddons class TestTermLog: - def test_simple(self): - sio = io.StringIO() - t = termlog.TermLog(outfile=sio) - t.configure(dump.Options(verbosity = 2), set([])) - t.log(log.LogEntry("one", "info")) - assert "one" in sio.getvalue() - t.log(log.LogEntry("two", "debug")) - assert "two" not in sio.getvalue() + @pytest.mark.usefixtures('capfd') + @pytest.mark.parametrize('outfile, expected_out, expected_err', [ + (None, ['one', 'three'], ['four']), + (sys.stdout, ['one', 'three', 'four'], []), + (sys.stderr, [], ['one', 'three', 'four']), + ]) + def test_output(self, outfile, expected_out, expected_err, capfd): + t = termlog.TermLog(outfile=outfile) + with taddons.context(options=Options(verbosity=2)) as tctx: + tctx.configure(t) + t.log(log.LogEntry("one", "info")) + t.log(log.LogEntry("two", "debug")) + t.log(log.LogEntry("three", "warn")) + t.log(log.LogEntry("four", "error")) + out, err = capfd.readouterr() + assert out.strip().splitlines() == expected_out + assert err.strip().splitlines() == expected_err diff --git a/test/mitmproxy/addons/test_view.py b/test/mitmproxy/addons/test_view.py index 96f213e2..da4ca007 100644 --- a/test/mitmproxy/addons/test_view.py +++ b/test/mitmproxy/addons/test_view.py @@ -19,15 +19,15 @@ class Options(options.Options): self, *, filter=None, - order=None, - order_reversed=False, - focus_follow=False, + console_order=None, + console_order_reversed=False, + console_focus_follow=False, **kwargs ): self.filter = filter - self.order = order - self.order_reversed = order_reversed - self.focus_follow = focus_follow + self.console_order = console_order + self.console_order_reversed = console_order_reversed + self.console_focus_follow = console_focus_follow super().__init__(**kwargs) @@ -42,7 +42,7 @@ def test_order_refresh(): tf = tflow.tflow(resp=True) with taddons.context(options=Options()) as tctx: - tctx.configure(v, order="time") + tctx.configure(v, console_order="time") v.add(tf) tf.request.timestamp_start = 1 assert not sargs @@ -73,12 +73,15 @@ def test_simple(): assert v.store_count() == 0 v.request(f) assert list(v) == [f] + assert v.get_by_id(f.id) + assert not v.get_by_id("nonexistent") # These all just call udpate v.error(f) v.response(f) v.intercept(f) v.resume(f) + v.kill(f) assert list(v) == [f] v.request(f) @@ -145,12 +148,12 @@ def test_order(): v.request(tft(method="put", start=4)) assert [i.request.timestamp_start for i in v] == [1, 2, 3, 4] - tctx.configure(v, order="method") + tctx.configure(v, console_order="method") assert [i.request.method for i in v] == ["GET", "GET", "PUT", "PUT"] v.set_reversed(True) assert [i.request.method for i in v] == ["PUT", "PUT", "GET", "GET"] - tctx.configure(v, order="time") + tctx.configure(v, console_order="time") assert [i.request.timestamp_start for i in v] == [4, 3, 2, 1] v.set_reversed(False) @@ -268,7 +271,7 @@ def test_signals(): def test_focus_follow(): v = view.View() with taddons.context(options=Options()) as tctx: - tctx.configure(v, focus_follow=True, filter="~m get") + tctx.configure(v, console_focus_follow=True, filter="~m get") v.add(tft(start=5)) assert v.focus.index == 0 @@ -381,12 +384,12 @@ def test_configure(): tctx.configure(v, filter="~q") tutils.raises("invalid interception filter", tctx.configure, v, filter="~~") - tctx.configure(v, order="method") - tutils.raises("unknown flow order", tctx.configure, v, order="no") + tctx.configure(v, console_order="method") + tutils.raises("unknown flow order", tctx.configure, v, console_order="no") - tctx.configure(v, order_reversed=True) + tctx.configure(v, console_order_reversed=True) - tctx.configure(v, order=None) + tctx.configure(v, console_order=None) - tctx.configure(v, focus_follow=True) + tctx.configure(v, console_focus_follow=True) assert v.focus_follow diff --git a/test/mitmproxy/console/test_pathedit.py b/test/mitmproxy/console/test_pathedit.py index b326ed6d..bd064e5f 100644 --- a/test/mitmproxy/console/test_pathedit.py +++ b/test/mitmproxy/console/test_pathedit.py @@ -3,7 +3,7 @@ from os.path import normpath from mitmproxy.tools.console import pathedit from mitmproxy.test import tutils -from mock import patch +from unittest.mock import patch class TestPathCompleter: diff --git a/test/mitmproxy/contentviews/test_api.py b/test/mitmproxy/contentviews/test_api.py index 8e6c3427..b4bd0106 100644 --- a/test/mitmproxy/contentviews/test_api.py +++ b/test/mitmproxy/contentviews/test_api.py @@ -1,4 +1,4 @@ -import mock +from unittest import mock from mitmproxy import contentviews from mitmproxy.exceptions import ContentViewException diff --git a/test/mitmproxy/mock_urwid.py b/test/mitmproxy/mock_urwid.py index 191210bf..9cc41abc 100644 --- a/test/mitmproxy/mock_urwid.py +++ b/test/mitmproxy/mock_urwid.py @@ -1,6 +1,7 @@ import os import sys -import mock +from unittest import mock + if os.name == "nt": m = mock.Mock() m.__version__ = "1.1.1" diff --git a/test/mitmproxy/net/http/http1/test_read.py b/test/mitmproxy/net/http/http1/test_read.py index 20997259..72f255b3 100644 --- a/test/mitmproxy/net/http/http1/test_read.py +++ b/test/mitmproxy/net/http/http1/test_read.py @@ -1,5 +1,5 @@ from io import BytesIO -from mock import Mock +from unittest.mock import Mock import pytest from mitmproxy import exceptions diff --git a/test/mitmproxy/net/http/test_cookies.py b/test/mitmproxy/net/http/test_cookies.py index 8c9c0c32..9c72ce3f 100644 --- a/test/mitmproxy/net/http/test_cookies.py +++ b/test/mitmproxy/net/http/test_cookies.py @@ -1,9 +1,9 @@ import time +from unittest import mock from mitmproxy.net.http import cookies from mitmproxy.test.tutils import raises -import mock cookie_pairs = [ [ diff --git a/test/mitmproxy/net/http/test_encoding.py b/test/mitmproxy/net/http/test_encoding.py index d8fa5e76..67d1a61b 100644 --- a/test/mitmproxy/net/http/test_encoding.py +++ b/test/mitmproxy/net/http/test_encoding.py @@ -1,4 +1,4 @@ -import mock +from unittest import mock import pytest from mitmproxy.net.http import encoding diff --git a/test/mitmproxy/net/test_tcp.py b/test/mitmproxy/net/test_tcp.py index fe44973b..ef7d3ea4 100644 --- a/test/mitmproxy/net/test_tcp.py +++ b/test/mitmproxy/net/test_tcp.py @@ -5,8 +5,8 @@ import socket import random import os import threading -import mock import pytest +from unittest import mock from OpenSSL import SSL from mitmproxy import certs diff --git a/test/mitmproxy/test_addonmanager.py b/test/mitmproxy/test_addonmanager.py index 7a50148e..17402e26 100644 --- a/test/mitmproxy/test_addonmanager.py +++ b/test/mitmproxy/test_addonmanager.py @@ -1,4 +1,7 @@ +import pytest + from mitmproxy import addonmanager +from mitmproxy import exceptions from mitmproxy import options from mitmproxy import master from mitmproxy import proxy @@ -7,10 +10,14 @@ from mitmproxy import proxy class TAddon: def __init__(self, name): self.name = name + self.noop_member = True def __repr__(self): return "Addon(%s)" % self.name + def noop(self): + pass + def test_simple(): o = options.Options() @@ -21,3 +28,8 @@ def test_simple(): assert not a.get("two") a.clear() assert not a.chain + + a.add(TAddon("one")) + a("noop") + with pytest.raises(exceptions.AddonError): + a("noop_member") diff --git a/test/mitmproxy/test_cmdline.py b/test/mitmproxy/test_cmdline.py index d2e0c8a5..96d5ae31 100644 --- a/test/mitmproxy/test_cmdline.py +++ b/test/mitmproxy/test_cmdline.py @@ -1,43 +1,5 @@ import argparse from mitmproxy.tools import cmdline -from mitmproxy.test import tutils - - -def test_parse_replace_hook(): - x = cmdline.parse_replace_hook("/foo/bar/voing") - assert x == ("foo", "bar", "voing") - - x = cmdline.parse_replace_hook("/foo/bar/vo/ing/") - assert x == ("foo", "bar", "vo/ing/") - - x = cmdline.parse_replace_hook("/bar/voing") - assert x == (".*", "bar", "voing") - - tutils.raises( - cmdline.ParseException, - cmdline.parse_replace_hook, - "/foo" - ) - tutils.raises( - "replacement regex", - cmdline.parse_replace_hook, - "patt/[/rep" - ) - tutils.raises( - "filter pattern", - cmdline.parse_replace_hook, - "/~/foo/rep" - ) - tutils.raises( - "empty clause", - cmdline.parse_replace_hook, - "//" - ) - - -def test_parse_setheaders(): - x = cmdline.parse_setheader("/foo/bar/voing") - assert x == ("foo", "bar", "voing") def test_common(): @@ -53,50 +15,6 @@ def test_common(): assert v["stickycookie"] == "foo" assert v["stickyauth"] == "foo" - opts.setheader = ["/foo/bar/voing"] - v = cmdline.get_common_options(opts) - assert v["setheaders"] == [("foo", "bar", "voing")] - - opts.setheader = ["//"] - tutils.raises( - "empty clause", - cmdline.get_common_options, - opts - ) - opts.setheader = [] - - opts.replace = ["/foo/bar/voing"] - v = cmdline.get_common_options(opts) - assert v["replacements"] == [("foo", "bar", "voing")] - - opts.replace = ["//"] - tutils.raises( - "empty clause", - cmdline.get_common_options, - opts - ) - - opts.replace = [] - opts.replace_file = [("/foo/bar/nonexistent")] - tutils.raises( - "could not read replace file", - cmdline.get_common_options, - opts - ) - - opts.replace_file = [("/~/bar/nonexistent")] - tutils.raises( - "filter pattern", - cmdline.get_common_options, - opts - ) - - p = tutils.test_data.path("mitmproxy/data/replace") - opts.replace_file = [("/foo/bar/%s" % p)] - v = cmdline.get_common_options(opts)["replacements"] - assert len(v) == 1 - assert v[0][2].strip() == b"replacecontents" - def test_mitmproxy(): ap = cmdline.mitmproxy() diff --git a/test/mitmproxy/test_controller.py b/test/mitmproxy/test_controller.py index 5cc36532..052affb8 100644 --- a/test/mitmproxy/test_controller.py +++ b/test/mitmproxy/test_controller.py @@ -1,6 +1,6 @@ from threading import Thread, Event -from mock import Mock +from unittest.mock import Mock from mitmproxy import controller import queue diff --git a/test/mitmproxy/test_flowfilter.py b/test/mitmproxy/test_flowfilter.py index 2d409994..1fb97126 100644 --- a/test/mitmproxy/test_flowfilter.py +++ b/test/mitmproxy/test_flowfilter.py @@ -1,6 +1,7 @@ import io +from unittest.mock import patch + from mitmproxy.test import tflow -from mock import patch from mitmproxy import flowfilter from . import tutils as ttutils diff --git a/test/mitmproxy/test_proxy.py b/test/mitmproxy/test_proxy.py index 0d63f147..6e0ef846 100644 --- a/test/mitmproxy/test_proxy.py +++ b/test/mitmproxy/test_proxy.py @@ -1,6 +1,6 @@ from mitmproxy.test import tflow import os -import mock +from unittest import mock import argparse from OpenSSL import SSL diff --git a/test/mitmproxy/test_server.py b/test/mitmproxy/test_server.py index 332d6138..1388a4d8 100644 --- a/test/mitmproxy/test_server.py +++ b/test/mitmproxy/test_server.py @@ -2,7 +2,7 @@ import os import socket import time -import mock +from unittest import mock from mitmproxy.test import tutils from mitmproxy import controller diff --git a/test/mitmproxy/test_tools_dump.py b/test/mitmproxy/test_tools_dump.py index 2e64d2d2..505f514b 100644 --- a/test/mitmproxy/test_tools_dump.py +++ b/test/mitmproxy/test_tools_dump.py @@ -1,10 +1,13 @@ import os +import pytest +from unittest import mock -from mitmproxy.tools import dump from mitmproxy import proxy -from mitmproxy.test import tutils from mitmproxy import log from mitmproxy import controller +from mitmproxy.tools import dump + +from mitmproxy.test import tutils from . import mastertest @@ -37,3 +40,17 @@ class TestDumpMaster(mastertest.MasterTest): ent.reply = controller.DummyReply() m.log(ent) assert m.has_errored + + @pytest.mark.parametrize("termlog", [False, True]) + def test_addons_termlog(self, termlog): + with mock.patch('sys.stdout'): + o = dump.Options() + m = dump.DumpMaster(o, proxy.DummyServer(), with_termlog=termlog) + assert (m.addons.get('termlog') is not None) == termlog + + @pytest.mark.parametrize("dumper", [False, True]) + def test_addons_dumper(self, dumper): + with mock.patch('sys.stdout'): + o = dump.Options() + m = dump.DumpMaster(o, proxy.DummyServer(), with_dumper=dumper) + assert (m.addons.get('dumper') is not None) == dumper diff --git a/test/mitmproxy/test_web_app.py b/test/mitmproxy/test_web_app.py index 7da5e4e9..00dc2c7c 100644 --- a/test/mitmproxy/test_web_app.py +++ b/test/mitmproxy/test_web_app.py @@ -1,15 +1,16 @@ import json as _json +from unittest import mock -import mock import tornado.testing +from tornado import httpclient +from tornado import websocket + from mitmproxy import exceptions from mitmproxy import proxy from mitmproxy import options from mitmproxy.test import tflow from mitmproxy.tools.web import app from mitmproxy.tools.web import master as webmaster -from tornado import httpclient -from tornado import websocket def json(resp: httpclient.HTTPResponse): @@ -18,7 +19,7 @@ def json(resp: httpclient.HTTPResponse): class TestApp(tornado.testing.AsyncHTTPTestCase): def get_app(self): - o = options.Options() + o = options.Options(http2=False) m = webmaster.WebMaster(o, proxy.DummyServer(), with_termlog=False) f = tflow.tflow(resp=True) f.id = "42" diff --git a/test/mitmproxy/test_web_master.py b/test/mitmproxy/test_web_master.py index 08dce8f3..3591284d 100644 --- a/test/mitmproxy/test_web_master.py +++ b/test/mitmproxy/test_web_master.py @@ -1,13 +1,16 @@ from mitmproxy.tools.web import master from mitmproxy import proxy from mitmproxy import options +from mitmproxy.proxy.config import ProxyConfig + from . import mastertest class TestWebMaster(mastertest.MasterTest): def mkmaster(self, **opts): o = options.Options(**opts) - return master.WebMaster(o, proxy.DummyServer(o)) + c = ProxyConfig(o) + return master.WebMaster(o, proxy.DummyServer(c)) def test_basic(self): m = self.mkmaster() diff --git a/test/mitmproxy/utils/test_typecheck.py b/test/mitmproxy/utils/test_typecheck.py index 0ed440eb..a21c65c9 100644 --- a/test/mitmproxy/utils/test_typecheck.py +++ b/test/mitmproxy/utils/test_typecheck.py @@ -1,7 +1,6 @@ import io import typing - -import mock +from unittest import mock import pytest from mitmproxy.utils import typecheck diff --git a/test/mitmproxy/utils/test_version_check.py b/test/mitmproxy/utils/test_version_check.py index 5c8d8c8c..d7929378 100644 --- a/test/mitmproxy/utils/test_version_check.py +++ b/test/mitmproxy/utils/test_version_check.py @@ -1,5 +1,5 @@ import io -import mock +from unittest import mock from mitmproxy.utils import version_check diff --git a/test/pathod/test_pathoc.py b/test/pathod/test_pathoc.py index 274e2be7..23e0f973 100644 --- a/test/pathod/test_pathoc.py +++ b/test/pathod/test_pathoc.py @@ -1,5 +1,5 @@ import io -from mock import Mock +from unittest.mock import Mock import pytest from mitmproxy.net import http diff --git a/test/pathod/test_pathoc_cmdline.py b/test/pathod/test_pathoc_cmdline.py index 3b54403f..710a816f 100644 --- a/test/pathod/test_pathoc_cmdline.py +++ b/test/pathod/test_pathoc_cmdline.py @@ -1,5 +1,5 @@ import io -import mock +from unittest import mock from pathod import pathoc_cmdline as cmdline diff --git a/test/pathod/test_pathod_cmdline.py b/test/pathod/test_pathod_cmdline.py index 290f5d9f..34baf491 100644 --- a/test/pathod/test_pathod_cmdline.py +++ b/test/pathod/test_pathod_cmdline.py @@ -1,4 +1,4 @@ -import mock +from unittest import mock from pathod import pathod_cmdline as cmdline diff --git a/test/pathod/test_protocols_http2.py b/test/pathod/test_protocols_http2.py index 8531887b..2f92dc54 100644 --- a/test/pathod/test_protocols_http2.py +++ b/test/pathod/test_protocols_http2.py @@ -1,4 +1,4 @@ -import mock +from unittest import mock import codecs import hyperframe diff --git a/web/src/js/filt/filt.js b/web/src/js/filt/filt.js index f2c5ded2..a1380e2b 100644 --- a/web/src/js/filt/filt.js +++ b/web/src/js/filt/filt.js @@ -59,94 +59,109 @@ module.exports = (function() { peg$c20 = ")", peg$c21 = { type: "literal", value: ")", description: "\")\"" }, peg$c22 = function(expr) { return binding(expr); }, - peg$c23 = "~a", - peg$c24 = { type: "literal", value: "~a", description: "\"~a\"" }, - peg$c25 = function() { return assetFilter; }, - peg$c26 = "~e", - peg$c27 = { type: "literal", value: "~e", description: "\"~e\"" }, - peg$c28 = function() { return errorFilter; }, - peg$c29 = "~http", - peg$c30 = { type: "literal", value: "~http", description: "\"~http\"" }, - peg$c31 = function() { return httpFilter; }, - peg$c32 = "~marked", - peg$c33 = { type: "literal", value: "~marked", description: "\"~marked\"" }, - peg$c34 = function() { return markedFilter; }, - peg$c35 = "~q", - peg$c36 = { type: "literal", value: "~q", description: "\"~q\"" }, - peg$c37 = function() { return noResponseFilter; }, - peg$c38 = "~s", - peg$c39 = { type: "literal", value: "~s", description: "\"~s\"" }, - peg$c40 = function() { return responseFilter; }, - peg$c41 = "~tcp", - peg$c42 = { type: "literal", value: "~tcp", description: "\"~tcp\"" }, - peg$c43 = function() { return tcpFilter; }, - peg$c44 = "true", - peg$c45 = { type: "literal", value: "true", description: "\"true\"" }, - peg$c46 = function() { return trueFilter; }, - peg$c47 = "false", - peg$c48 = { type: "literal", value: "false", description: "\"false\"" }, - peg$c49 = function() { return falseFilter; }, - peg$c50 = "~c", - peg$c51 = { type: "literal", value: "~c", description: "\"~c\"" }, - peg$c52 = function(s) { return responseCode(s); }, - peg$c53 = "~d", - peg$c54 = { type: "literal", value: "~d", description: "\"~d\"" }, - peg$c55 = function(s) { return domain(s); }, - peg$c56 = "~h", - peg$c57 = { type: "literal", value: "~h", description: "\"~h\"" }, - peg$c58 = function(s) { return header(s); }, - peg$c59 = "~hq", - peg$c60 = { type: "literal", value: "~hq", description: "\"~hq\"" }, - peg$c61 = function(s) { return requestHeader(s); }, - peg$c62 = "~hs", - peg$c63 = { type: "literal", value: "~hs", description: "\"~hs\"" }, - peg$c64 = function(s) { return responseHeader(s); }, + peg$c23 = "true", + peg$c24 = { type: "literal", value: "true", description: "\"true\"" }, + peg$c25 = function() { return trueFilter; }, + peg$c26 = "false", + peg$c27 = { type: "literal", value: "false", description: "\"false\"" }, + peg$c28 = function() { return falseFilter; }, + peg$c29 = "~a", + peg$c30 = { type: "literal", value: "~a", description: "\"~a\"" }, + peg$c31 = function() { return assetFilter; }, + peg$c32 = "~b", + peg$c33 = { type: "literal", value: "~b", description: "\"~b\"" }, + peg$c34 = function(s) { return body(s); }, + peg$c35 = "~bq", + peg$c36 = { type: "literal", value: "~bq", description: "\"~bq\"" }, + peg$c37 = function(s) { return requestBody(s); }, + peg$c38 = "~bs", + peg$c39 = { type: "literal", value: "~bs", description: "\"~bs\"" }, + peg$c40 = function(s) { return responseBody(s); }, + peg$c41 = "~c", + peg$c42 = { type: "literal", value: "~c", description: "\"~c\"" }, + peg$c43 = function(s) { return responseCode(s); }, + peg$c44 = "~d", + peg$c45 = { type: "literal", value: "~d", description: "\"~d\"" }, + peg$c46 = function(s) { return domain(s); }, + peg$c47 = "~dst", + peg$c48 = { type: "literal", value: "~dst", description: "\"~dst\"" }, + peg$c49 = function(s) { return destination(s); }, + peg$c50 = "~e", + peg$c51 = { type: "literal", value: "~e", description: "\"~e\"" }, + peg$c52 = function() { return errorFilter; }, + peg$c53 = "~h", + peg$c54 = { type: "literal", value: "~h", description: "\"~h\"" }, + peg$c55 = function(s) { return header(s); }, + peg$c56 = "~hq", + peg$c57 = { type: "literal", value: "~hq", description: "\"~hq\"" }, + peg$c58 = function(s) { return requestHeader(s); }, + peg$c59 = "~hs", + peg$c60 = { type: "literal", value: "~hs", description: "\"~hs\"" }, + peg$c61 = function(s) { return responseHeader(s); }, + peg$c62 = "~http", + peg$c63 = { type: "literal", value: "~http", description: "\"~http\"" }, + peg$c64 = function() { return httpFilter; }, peg$c65 = "~m", peg$c66 = { type: "literal", value: "~m", description: "\"~m\"" }, peg$c67 = function(s) { return method(s); }, - peg$c68 = "~t", - peg$c69 = { type: "literal", value: "~t", description: "\"~t\"" }, - peg$c70 = function(s) { return contentType(s); }, - peg$c71 = "~tq", - peg$c72 = { type: "literal", value: "~tq", description: "\"~tq\"" }, - peg$c73 = function(s) { return requestContentType(s); }, - peg$c74 = "~ts", - peg$c75 = { type: "literal", value: "~ts", description: "\"~ts\"" }, - peg$c76 = function(s) { return responseContentType(s); }, - peg$c77 = "~u", - peg$c78 = { type: "literal", value: "~u", description: "\"~u\"" }, - peg$c79 = function(s) { return url(s); }, - peg$c80 = { type: "other", description: "integer" }, - peg$c81 = /^['"]/, - peg$c82 = { type: "class", value: "['\"]", description: "['\"]" }, - peg$c83 = /^[0-9]/, - peg$c84 = { type: "class", value: "[0-9]", description: "[0-9]" }, - peg$c85 = function(digits) { return parseInt(digits.join(""), 10); }, - peg$c86 = { type: "other", description: "string" }, - peg$c87 = "\"", - peg$c88 = { type: "literal", value: "\"", description: "\"\\\"\"" }, - peg$c89 = function(chars) { return chars.join(""); }, - peg$c90 = "'", - peg$c91 = { type: "literal", value: "'", description: "\"'\"" }, - peg$c92 = /^["\\]/, - peg$c93 = { type: "class", value: "[\"\\\\]", description: "[\"\\\\]" }, - peg$c94 = { type: "any", description: "any character" }, - peg$c95 = function(char) { return char; }, - peg$c96 = "\\", - peg$c97 = { type: "literal", value: "\\", description: "\"\\\\\"" }, - peg$c98 = /^['\\]/, - peg$c99 = { type: "class", value: "['\\\\]", description: "['\\\\]" }, - peg$c100 = /^['"\\]/, - peg$c101 = { type: "class", value: "['\"\\\\]", description: "['\"\\\\]" }, - peg$c102 = "n", - peg$c103 = { type: "literal", value: "n", description: "\"n\"" }, - peg$c104 = function() { return "\n"; }, - peg$c105 = "r", - peg$c106 = { type: "literal", value: "r", description: "\"r\"" }, - peg$c107 = function() { return "\r"; }, - peg$c108 = "t", - peg$c109 = { type: "literal", value: "t", description: "\"t\"" }, - peg$c110 = function() { return "\t"; }, + peg$c68 = "~marked", + peg$c69 = { type: "literal", value: "~marked", description: "\"~marked\"" }, + peg$c70 = function() { return markedFilter; }, + peg$c71 = "~q", + peg$c72 = { type: "literal", value: "~q", description: "\"~q\"" }, + peg$c73 = function() { return noResponseFilter; }, + peg$c74 = "~src", + peg$c75 = { type: "literal", value: "~src", description: "\"~src\"" }, + peg$c76 = function(s) { return source(s); }, + peg$c77 = "~s", + peg$c78 = { type: "literal", value: "~s", description: "\"~s\"" }, + peg$c79 = function() { return responseFilter; }, + peg$c80 = "~t", + peg$c81 = { type: "literal", value: "~t", description: "\"~t\"" }, + peg$c82 = function(s) { return contentType(s); }, + peg$c83 = "~tcp", + peg$c84 = { type: "literal", value: "~tcp", description: "\"~tcp\"" }, + peg$c85 = function() { return tcpFilter; }, + peg$c86 = "~tq", + peg$c87 = { type: "literal", value: "~tq", description: "\"~tq\"" }, + peg$c88 = function(s) { return requestContentType(s); }, + peg$c89 = "~ts", + peg$c90 = { type: "literal", value: "~ts", description: "\"~ts\"" }, + peg$c91 = function(s) { return responseContentType(s); }, + peg$c92 = "~u", + peg$c93 = { type: "literal", value: "~u", description: "\"~u\"" }, + peg$c94 = function(s) { return url(s); }, + peg$c95 = { type: "other", description: "integer" }, + peg$c96 = /^['"]/, + peg$c97 = { type: "class", value: "['\"]", description: "['\"]" }, + peg$c98 = /^[0-9]/, + peg$c99 = { type: "class", value: "[0-9]", description: "[0-9]" }, + peg$c100 = function(digits) { return parseInt(digits.join(""), 10); }, + peg$c101 = { type: "other", description: "string" }, + peg$c102 = "\"", + peg$c103 = { type: "literal", value: "\"", description: "\"\\\"\"" }, + peg$c104 = function(chars) { return chars.join(""); }, + peg$c105 = "'", + peg$c106 = { type: "literal", value: "'", description: "\"'\"" }, + peg$c107 = /^["\\]/, + peg$c108 = { type: "class", value: "[\"\\\\]", description: "[\"\\\\]" }, + peg$c109 = { type: "any", description: "any character" }, + peg$c110 = function(char) { return char; }, + peg$c111 = "\\", + peg$c112 = { type: "literal", value: "\\", description: "\"\\\\\"" }, + peg$c113 = /^['\\]/, + peg$c114 = { type: "class", value: "['\\\\]", description: "['\\\\]" }, + peg$c115 = /^['"\\]/, + peg$c116 = { type: "class", value: "['\"\\\\]", description: "['\"\\\\]" }, + peg$c117 = "n", + peg$c118 = { type: "literal", value: "n", description: "\"n\"" }, + peg$c119 = function() { return "\n"; }, + peg$c120 = "r", + peg$c121 = { type: "literal", value: "r", description: "\"r\"" }, + peg$c122 = function() { return "\r"; }, + peg$c123 = "t", + peg$c124 = { type: "literal", value: "t", description: "\"t\"" }, + peg$c125 = function() { return "\t"; }, peg$currPos = 0, peg$savedPos = 0, @@ -655,288 +670,57 @@ module.exports = (function() { } function peg$parseExpr() { - var s0; - - s0 = peg$parseNullaryExpr(); - if (s0 === peg$FAILED) { - s0 = peg$parseUnaryExpr(); - } - - return s0; - } - - function peg$parseNullaryExpr() { - var s0, s1; - - s0 = peg$parseBooleanLiteral(); - if (s0 === peg$FAILED) { - s0 = peg$currPos; - if (input.substr(peg$currPos, 2) === peg$c23) { - s1 = peg$c23; - peg$currPos += 2; - } else { - s1 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c24); } - } - if (s1 !== peg$FAILED) { - peg$savedPos = s0; - s1 = peg$c25(); - } - s0 = s1; - if (s0 === peg$FAILED) { - s0 = peg$currPos; - if (input.substr(peg$currPos, 2) === peg$c26) { - s1 = peg$c26; - peg$currPos += 2; - } else { - s1 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c27); } - } - if (s1 !== peg$FAILED) { - peg$savedPos = s0; - s1 = peg$c28(); - } - s0 = s1; - if (s0 === peg$FAILED) { - s0 = peg$currPos; - if (input.substr(peg$currPos, 5) === peg$c29) { - s1 = peg$c29; - peg$currPos += 5; - } else { - s1 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c30); } - } - if (s1 !== peg$FAILED) { - peg$savedPos = s0; - s1 = peg$c31(); - } - s0 = s1; - if (s0 === peg$FAILED) { - s0 = peg$currPos; - if (input.substr(peg$currPos, 7) === peg$c32) { - s1 = peg$c32; - peg$currPos += 7; - } else { - s1 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c33); } - } - if (s1 !== peg$FAILED) { - peg$savedPos = s0; - s1 = peg$c34(); - } - s0 = s1; - if (s0 === peg$FAILED) { - s0 = peg$currPos; - if (input.substr(peg$currPos, 2) === peg$c35) { - s1 = peg$c35; - peg$currPos += 2; - } else { - s1 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c36); } - } - if (s1 !== peg$FAILED) { - peg$savedPos = s0; - s1 = peg$c37(); - } - s0 = s1; - if (s0 === peg$FAILED) { - s0 = peg$currPos; - if (input.substr(peg$currPos, 2) === peg$c38) { - s1 = peg$c38; - peg$currPos += 2; - } else { - s1 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c39); } - } - if (s1 !== peg$FAILED) { - peg$savedPos = s0; - s1 = peg$c40(); - } - s0 = s1; - if (s0 === peg$FAILED) { - s0 = peg$currPos; - if (input.substr(peg$currPos, 4) === peg$c41) { - s1 = peg$c41; - peg$currPos += 4; - } else { - s1 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c42); } - } - if (s1 !== peg$FAILED) { - peg$savedPos = s0; - s1 = peg$c43(); - } - s0 = s1; - } - } - } - } - } - } - } - - return s0; - } - - function peg$parseBooleanLiteral() { - var s0, s1; + var s0, s1, s2, s3; s0 = peg$currPos; - if (input.substr(peg$currPos, 4) === peg$c44) { - s1 = peg$c44; + if (input.substr(peg$currPos, 4) === peg$c23) { + s1 = peg$c23; peg$currPos += 4; } else { s1 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c45); } + if (peg$silentFails === 0) { peg$fail(peg$c24); } } if (s1 !== peg$FAILED) { peg$savedPos = s0; - s1 = peg$c46(); + s1 = peg$c25(); } s0 = s1; if (s0 === peg$FAILED) { s0 = peg$currPos; - if (input.substr(peg$currPos, 5) === peg$c47) { - s1 = peg$c47; + if (input.substr(peg$currPos, 5) === peg$c26) { + s1 = peg$c26; peg$currPos += 5; } else { s1 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c48); } + if (peg$silentFails === 0) { peg$fail(peg$c27); } } if (s1 !== peg$FAILED) { peg$savedPos = s0; - s1 = peg$c49(); + s1 = peg$c28(); } s0 = s1; - } - - return s0; - } - - function peg$parseUnaryExpr() { - var s0, s1, s2, s3; - - s0 = peg$currPos; - if (input.substr(peg$currPos, 2) === peg$c50) { - s1 = peg$c50; - peg$currPos += 2; - } else { - s1 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c51); } - } - if (s1 !== peg$FAILED) { - s2 = []; - s3 = peg$parsews(); - if (s3 !== peg$FAILED) { - while (s3 !== peg$FAILED) { - s2.push(s3); - s3 = peg$parsews(); - } - } else { - s2 = peg$FAILED; - } - if (s2 !== peg$FAILED) { - s3 = peg$parseIntegerLiteral(); - if (s3 !== peg$FAILED) { - peg$savedPos = s0; - s1 = peg$c52(s3); - s0 = s1; - } else { - peg$currPos = s0; - s0 = peg$FAILED; - } - } else { - peg$currPos = s0; - s0 = peg$FAILED; - } - } else { - peg$currPos = s0; - s0 = peg$FAILED; - } - if (s0 === peg$FAILED) { - s0 = peg$currPos; - if (input.substr(peg$currPos, 2) === peg$c53) { - s1 = peg$c53; - peg$currPos += 2; - } else { - s1 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c54); } - } - if (s1 !== peg$FAILED) { - s2 = []; - s3 = peg$parsews(); - if (s3 !== peg$FAILED) { - while (s3 !== peg$FAILED) { - s2.push(s3); - s3 = peg$parsews(); - } - } else { - s2 = peg$FAILED; - } - if (s2 !== peg$FAILED) { - s3 = peg$parseStringLiteral(); - if (s3 !== peg$FAILED) { - peg$savedPos = s0; - s1 = peg$c55(s3); - s0 = s1; - } else { - peg$currPos = s0; - s0 = peg$FAILED; - } - } else { - peg$currPos = s0; - s0 = peg$FAILED; - } - } else { - peg$currPos = s0; - s0 = peg$FAILED; - } if (s0 === peg$FAILED) { s0 = peg$currPos; - if (input.substr(peg$currPos, 2) === peg$c56) { - s1 = peg$c56; + if (input.substr(peg$currPos, 2) === peg$c29) { + s1 = peg$c29; peg$currPos += 2; } else { s1 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c57); } + if (peg$silentFails === 0) { peg$fail(peg$c30); } } if (s1 !== peg$FAILED) { - s2 = []; - s3 = peg$parsews(); - if (s3 !== peg$FAILED) { - while (s3 !== peg$FAILED) { - s2.push(s3); - s3 = peg$parsews(); - } - } else { - s2 = peg$FAILED; - } - if (s2 !== peg$FAILED) { - s3 = peg$parseStringLiteral(); - if (s3 !== peg$FAILED) { - peg$savedPos = s0; - s1 = peg$c58(s3); - s0 = s1; - } else { - peg$currPos = s0; - s0 = peg$FAILED; - } - } else { - peg$currPos = s0; - s0 = peg$FAILED; - } - } else { - peg$currPos = s0; - s0 = peg$FAILED; + peg$savedPos = s0; + s1 = peg$c31(); } + s0 = s1; if (s0 === peg$FAILED) { s0 = peg$currPos; - if (input.substr(peg$currPos, 3) === peg$c59) { - s1 = peg$c59; - peg$currPos += 3; + if (input.substr(peg$currPos, 2) === peg$c32) { + s1 = peg$c32; + peg$currPos += 2; } else { s1 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c60); } + if (peg$silentFails === 0) { peg$fail(peg$c33); } } if (s1 !== peg$FAILED) { s2 = []; @@ -953,7 +737,7 @@ module.exports = (function() { s3 = peg$parseStringLiteral(); if (s3 !== peg$FAILED) { peg$savedPos = s0; - s1 = peg$c61(s3); + s1 = peg$c34(s3); s0 = s1; } else { peg$currPos = s0; @@ -969,12 +753,12 @@ module.exports = (function() { } if (s0 === peg$FAILED) { s0 = peg$currPos; - if (input.substr(peg$currPos, 3) === peg$c62) { - s1 = peg$c62; + if (input.substr(peg$currPos, 3) === peg$c35) { + s1 = peg$c35; peg$currPos += 3; } else { s1 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c63); } + if (peg$silentFails === 0) { peg$fail(peg$c36); } } if (s1 !== peg$FAILED) { s2 = []; @@ -991,7 +775,7 @@ module.exports = (function() { s3 = peg$parseStringLiteral(); if (s3 !== peg$FAILED) { peg$savedPos = s0; - s1 = peg$c64(s3); + s1 = peg$c37(s3); s0 = s1; } else { peg$currPos = s0; @@ -1007,12 +791,12 @@ module.exports = (function() { } if (s0 === peg$FAILED) { s0 = peg$currPos; - if (input.substr(peg$currPos, 2) === peg$c65) { - s1 = peg$c65; - peg$currPos += 2; + if (input.substr(peg$currPos, 3) === peg$c38) { + s1 = peg$c38; + peg$currPos += 3; } else { s1 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c66); } + if (peg$silentFails === 0) { peg$fail(peg$c39); } } if (s1 !== peg$FAILED) { s2 = []; @@ -1029,7 +813,7 @@ module.exports = (function() { s3 = peg$parseStringLiteral(); if (s3 !== peg$FAILED) { peg$savedPos = s0; - s1 = peg$c67(s3); + s1 = peg$c40(s3); s0 = s1; } else { peg$currPos = s0; @@ -1045,12 +829,12 @@ module.exports = (function() { } if (s0 === peg$FAILED) { s0 = peg$currPos; - if (input.substr(peg$currPos, 2) === peg$c68) { - s1 = peg$c68; + if (input.substr(peg$currPos, 2) === peg$c41) { + s1 = peg$c41; peg$currPos += 2; } else { s1 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c69); } + if (peg$silentFails === 0) { peg$fail(peg$c42); } } if (s1 !== peg$FAILED) { s2 = []; @@ -1064,10 +848,10 @@ module.exports = (function() { s2 = peg$FAILED; } if (s2 !== peg$FAILED) { - s3 = peg$parseStringLiteral(); + s3 = peg$parseIntegerLiteral(); if (s3 !== peg$FAILED) { peg$savedPos = s0; - s1 = peg$c70(s3); + s1 = peg$c43(s3); s0 = s1; } else { peg$currPos = s0; @@ -1083,12 +867,12 @@ module.exports = (function() { } if (s0 === peg$FAILED) { s0 = peg$currPos; - if (input.substr(peg$currPos, 3) === peg$c71) { - s1 = peg$c71; - peg$currPos += 3; + if (input.substr(peg$currPos, 2) === peg$c44) { + s1 = peg$c44; + peg$currPos += 2; } else { s1 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c72); } + if (peg$silentFails === 0) { peg$fail(peg$c45); } } if (s1 !== peg$FAILED) { s2 = []; @@ -1105,7 +889,7 @@ module.exports = (function() { s3 = peg$parseStringLiteral(); if (s3 !== peg$FAILED) { peg$savedPos = s0; - s1 = peg$c73(s3); + s1 = peg$c46(s3); s0 = s1; } else { peg$currPos = s0; @@ -1121,12 +905,12 @@ module.exports = (function() { } if (s0 === peg$FAILED) { s0 = peg$currPos; - if (input.substr(peg$currPos, 3) === peg$c74) { - s1 = peg$c74; - peg$currPos += 3; + if (input.substr(peg$currPos, 4) === peg$c47) { + s1 = peg$c47; + peg$currPos += 4; } else { s1 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c75); } + if (peg$silentFails === 0) { peg$fail(peg$c48); } } if (s1 !== peg$FAILED) { s2 = []; @@ -1143,7 +927,7 @@ module.exports = (function() { s3 = peg$parseStringLiteral(); if (s3 !== peg$FAILED) { peg$savedPos = s0; - s1 = peg$c76(s3); + s1 = peg$c49(s3); s0 = s1; } else { peg$currPos = s0; @@ -1159,30 +943,48 @@ module.exports = (function() { } if (s0 === peg$FAILED) { s0 = peg$currPos; - if (input.substr(peg$currPos, 2) === peg$c77) { - s1 = peg$c77; + if (input.substr(peg$currPos, 2) === peg$c50) { + s1 = peg$c50; peg$currPos += 2; } else { s1 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c78); } + if (peg$silentFails === 0) { peg$fail(peg$c51); } } if (s1 !== peg$FAILED) { - s2 = []; - s3 = peg$parsews(); - if (s3 !== peg$FAILED) { - while (s3 !== peg$FAILED) { - s2.push(s3); - s3 = peg$parsews(); - } + peg$savedPos = s0; + s1 = peg$c52(); + } + s0 = s1; + if (s0 === peg$FAILED) { + s0 = peg$currPos; + if (input.substr(peg$currPos, 2) === peg$c53) { + s1 = peg$c53; + peg$currPos += 2; } else { - s2 = peg$FAILED; + s1 = peg$FAILED; + if (peg$silentFails === 0) { peg$fail(peg$c54); } } - if (s2 !== peg$FAILED) { - s3 = peg$parseStringLiteral(); + if (s1 !== peg$FAILED) { + s2 = []; + s3 = peg$parsews(); if (s3 !== peg$FAILED) { - peg$savedPos = s0; - s1 = peg$c79(s3); - s0 = s1; + while (s3 !== peg$FAILED) { + s2.push(s3); + s3 = peg$parsews(); + } + } else { + s2 = peg$FAILED; + } + if (s2 !== peg$FAILED) { + s3 = peg$parseStringLiteral(); + if (s3 !== peg$FAILED) { + peg$savedPos = s0; + s1 = peg$c55(s3); + s0 = s1; + } else { + peg$currPos = s0; + s0 = peg$FAILED; + } } else { peg$currPos = s0; s0 = peg$FAILED; @@ -1191,18 +993,402 @@ module.exports = (function() { peg$currPos = s0; s0 = peg$FAILED; } - } else { - peg$currPos = s0; - s0 = peg$FAILED; - } - if (s0 === peg$FAILED) { - s0 = peg$currPos; - s1 = peg$parseStringLiteral(); - if (s1 !== peg$FAILED) { - peg$savedPos = s0; - s1 = peg$c79(s1); + if (s0 === peg$FAILED) { + s0 = peg$currPos; + if (input.substr(peg$currPos, 3) === peg$c56) { + s1 = peg$c56; + peg$currPos += 3; + } else { + s1 = peg$FAILED; + if (peg$silentFails === 0) { peg$fail(peg$c57); } + } + if (s1 !== peg$FAILED) { + s2 = []; + s3 = peg$parsews(); + if (s3 !== peg$FAILED) { + while (s3 !== peg$FAILED) { + s2.push(s3); + s3 = peg$parsews(); + } + } else { + s2 = peg$FAILED; + } + if (s2 !== peg$FAILED) { + s3 = peg$parseStringLiteral(); + if (s3 !== peg$FAILED) { + peg$savedPos = s0; + s1 = peg$c58(s3); + s0 = s1; + } else { + peg$currPos = s0; + s0 = peg$FAILED; + } + } else { + peg$currPos = s0; + s0 = peg$FAILED; + } + } else { + peg$currPos = s0; + s0 = peg$FAILED; + } + if (s0 === peg$FAILED) { + s0 = peg$currPos; + if (input.substr(peg$currPos, 3) === peg$c59) { + s1 = peg$c59; + peg$currPos += 3; + } else { + s1 = peg$FAILED; + if (peg$silentFails === 0) { peg$fail(peg$c60); } + } + if (s1 !== peg$FAILED) { + s2 = []; + s3 = peg$parsews(); + if (s3 !== peg$FAILED) { + while (s3 !== peg$FAILED) { + s2.push(s3); + s3 = peg$parsews(); + } + } else { + s2 = peg$FAILED; + } + if (s2 !== peg$FAILED) { + s3 = peg$parseStringLiteral(); + if (s3 !== peg$FAILED) { + peg$savedPos = s0; + s1 = peg$c61(s3); + s0 = s1; + } else { + peg$currPos = s0; + s0 = peg$FAILED; + } + } else { + peg$currPos = s0; + s0 = peg$FAILED; + } + } else { + peg$currPos = s0; + s0 = peg$FAILED; + } + if (s0 === peg$FAILED) { + s0 = peg$currPos; + if (input.substr(peg$currPos, 5) === peg$c62) { + s1 = peg$c62; + peg$currPos += 5; + } else { + s1 = peg$FAILED; + if (peg$silentFails === 0) { peg$fail(peg$c63); } + } + if (s1 !== peg$FAILED) { + peg$savedPos = s0; + s1 = peg$c64(); + } + s0 = s1; + if (s0 === peg$FAILED) { + s0 = peg$currPos; + if (input.substr(peg$currPos, 2) === peg$c65) { + s1 = peg$c65; + peg$currPos += 2; + } else { + s1 = peg$FAILED; + if (peg$silentFails === 0) { peg$fail(peg$c66); } + } + if (s1 !== peg$FAILED) { + s2 = []; + s3 = peg$parsews(); + if (s3 !== peg$FAILED) { + while (s3 !== peg$FAILED) { + s2.push(s3); + s3 = peg$parsews(); + } + } else { + s2 = peg$FAILED; + } + if (s2 !== peg$FAILED) { + s3 = peg$parseStringLiteral(); + if (s3 !== peg$FAILED) { + peg$savedPos = s0; + s1 = peg$c67(s3); + s0 = s1; + } else { + peg$currPos = s0; + s0 = peg$FAILED; + } + } else { + peg$currPos = s0; + s0 = peg$FAILED; + } + } else { + peg$currPos = s0; + s0 = peg$FAILED; + } + if (s0 === peg$FAILED) { + s0 = peg$currPos; + if (input.substr(peg$currPos, 7) === peg$c68) { + s1 = peg$c68; + peg$currPos += 7; + } else { + s1 = peg$FAILED; + if (peg$silentFails === 0) { peg$fail(peg$c69); } + } + if (s1 !== peg$FAILED) { + peg$savedPos = s0; + s1 = peg$c70(); + } + s0 = s1; + if (s0 === peg$FAILED) { + s0 = peg$currPos; + if (input.substr(peg$currPos, 2) === peg$c71) { + s1 = peg$c71; + peg$currPos += 2; + } else { + s1 = peg$FAILED; + if (peg$silentFails === 0) { peg$fail(peg$c72); } + } + if (s1 !== peg$FAILED) { + peg$savedPos = s0; + s1 = peg$c73(); + } + s0 = s1; + if (s0 === peg$FAILED) { + s0 = peg$currPos; + if (input.substr(peg$currPos, 4) === peg$c74) { + s1 = peg$c74; + peg$currPos += 4; + } else { + s1 = peg$FAILED; + if (peg$silentFails === 0) { peg$fail(peg$c75); } + } + if (s1 !== peg$FAILED) { + s2 = []; + s3 = peg$parsews(); + if (s3 !== peg$FAILED) { + while (s3 !== peg$FAILED) { + s2.push(s3); + s3 = peg$parsews(); + } + } else { + s2 = peg$FAILED; + } + if (s2 !== peg$FAILED) { + s3 = peg$parseStringLiteral(); + if (s3 !== peg$FAILED) { + peg$savedPos = s0; + s1 = peg$c76(s3); + s0 = s1; + } else { + peg$currPos = s0; + s0 = peg$FAILED; + } + } else { + peg$currPos = s0; + s0 = peg$FAILED; + } + } else { + peg$currPos = s0; + s0 = peg$FAILED; + } + if (s0 === peg$FAILED) { + s0 = peg$currPos; + if (input.substr(peg$currPos, 2) === peg$c77) { + s1 = peg$c77; + peg$currPos += 2; + } else { + s1 = peg$FAILED; + if (peg$silentFails === 0) { peg$fail(peg$c78); } + } + if (s1 !== peg$FAILED) { + peg$savedPos = s0; + s1 = peg$c79(); + } + s0 = s1; + if (s0 === peg$FAILED) { + s0 = peg$currPos; + if (input.substr(peg$currPos, 2) === peg$c80) { + s1 = peg$c80; + peg$currPos += 2; + } else { + s1 = peg$FAILED; + if (peg$silentFails === 0) { peg$fail(peg$c81); } + } + if (s1 !== peg$FAILED) { + s2 = []; + s3 = peg$parsews(); + if (s3 !== peg$FAILED) { + while (s3 !== peg$FAILED) { + s2.push(s3); + s3 = peg$parsews(); + } + } else { + s2 = peg$FAILED; + } + if (s2 !== peg$FAILED) { + s3 = peg$parseStringLiteral(); + if (s3 !== peg$FAILED) { + peg$savedPos = s0; + s1 = peg$c82(s3); + s0 = s1; + } else { + peg$currPos = s0; + s0 = peg$FAILED; + } + } else { + peg$currPos = s0; + s0 = peg$FAILED; + } + } else { + peg$currPos = s0; + s0 = peg$FAILED; + } + if (s0 === peg$FAILED) { + s0 = peg$currPos; + if (input.substr(peg$currPos, 4) === peg$c83) { + s1 = peg$c83; + peg$currPos += 4; + } else { + s1 = peg$FAILED; + if (peg$silentFails === 0) { peg$fail(peg$c84); } + } + if (s1 !== peg$FAILED) { + peg$savedPos = s0; + s1 = peg$c85(); + } + s0 = s1; + if (s0 === peg$FAILED) { + s0 = peg$currPos; + if (input.substr(peg$currPos, 3) === peg$c86) { + s1 = peg$c86; + peg$currPos += 3; + } else { + s1 = peg$FAILED; + if (peg$silentFails === 0) { peg$fail(peg$c87); } + } + if (s1 !== peg$FAILED) { + s2 = []; + s3 = peg$parsews(); + if (s3 !== peg$FAILED) { + while (s3 !== peg$FAILED) { + s2.push(s3); + s3 = peg$parsews(); + } + } else { + s2 = peg$FAILED; + } + if (s2 !== peg$FAILED) { + s3 = peg$parseStringLiteral(); + if (s3 !== peg$FAILED) { + peg$savedPos = s0; + s1 = peg$c88(s3); + s0 = s1; + } else { + peg$currPos = s0; + s0 = peg$FAILED; + } + } else { + peg$currPos = s0; + s0 = peg$FAILED; + } + } else { + peg$currPos = s0; + s0 = peg$FAILED; + } + if (s0 === peg$FAILED) { + s0 = peg$currPos; + if (input.substr(peg$currPos, 3) === peg$c89) { + s1 = peg$c89; + peg$currPos += 3; + } else { + s1 = peg$FAILED; + if (peg$silentFails === 0) { peg$fail(peg$c90); } + } + if (s1 !== peg$FAILED) { + s2 = []; + s3 = peg$parsews(); + if (s3 !== peg$FAILED) { + while (s3 !== peg$FAILED) { + s2.push(s3); + s3 = peg$parsews(); + } + } else { + s2 = peg$FAILED; + } + if (s2 !== peg$FAILED) { + s3 = peg$parseStringLiteral(); + if (s3 !== peg$FAILED) { + peg$savedPos = s0; + s1 = peg$c91(s3); + s0 = s1; + } else { + peg$currPos = s0; + s0 = peg$FAILED; + } + } else { + peg$currPos = s0; + s0 = peg$FAILED; + } + } else { + peg$currPos = s0; + s0 = peg$FAILED; + } + if (s0 === peg$FAILED) { + s0 = peg$currPos; + if (input.substr(peg$currPos, 2) === peg$c92) { + s1 = peg$c92; + peg$currPos += 2; + } else { + s1 = peg$FAILED; + if (peg$silentFails === 0) { peg$fail(peg$c93); } + } + if (s1 !== peg$FAILED) { + s2 = []; + s3 = peg$parsews(); + if (s3 !== peg$FAILED) { + while (s3 !== peg$FAILED) { + s2.push(s3); + s3 = peg$parsews(); + } + } else { + s2 = peg$FAILED; + } + if (s2 !== peg$FAILED) { + s3 = peg$parseStringLiteral(); + if (s3 !== peg$FAILED) { + peg$savedPos = s0; + s1 = peg$c94(s3); + s0 = s1; + } else { + peg$currPos = s0; + s0 = peg$FAILED; + } + } else { + peg$currPos = s0; + s0 = peg$FAILED; + } + } else { + peg$currPos = s0; + s0 = peg$FAILED; + } + if (s0 === peg$FAILED) { + s0 = peg$currPos; + s1 = peg$parseStringLiteral(); + if (s1 !== peg$FAILED) { + peg$savedPos = s0; + s1 = peg$c94(s1); + } + s0 = s1; + } + } + } + } + } + } + } + } + } + } + } + } + } } - s0 = s1; } } } @@ -1222,53 +1408,53 @@ module.exports = (function() { peg$silentFails++; s0 = peg$currPos; - if (peg$c81.test(input.charAt(peg$currPos))) { + if (peg$c96.test(input.charAt(peg$currPos))) { s1 = input.charAt(peg$currPos); peg$currPos++; } else { s1 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c82); } + if (peg$silentFails === 0) { peg$fail(peg$c97); } } if (s1 === peg$FAILED) { s1 = null; } if (s1 !== peg$FAILED) { s2 = []; - if (peg$c83.test(input.charAt(peg$currPos))) { + if (peg$c98.test(input.charAt(peg$currPos))) { s3 = input.charAt(peg$currPos); peg$currPos++; } else { s3 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c84); } + if (peg$silentFails === 0) { peg$fail(peg$c99); } } if (s3 !== peg$FAILED) { while (s3 !== peg$FAILED) { s2.push(s3); - if (peg$c83.test(input.charAt(peg$currPos))) { + if (peg$c98.test(input.charAt(peg$currPos))) { s3 = input.charAt(peg$currPos); peg$currPos++; } else { s3 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c84); } + if (peg$silentFails === 0) { peg$fail(peg$c99); } } } } else { s2 = peg$FAILED; } if (s2 !== peg$FAILED) { - if (peg$c81.test(input.charAt(peg$currPos))) { + if (peg$c96.test(input.charAt(peg$currPos))) { s3 = input.charAt(peg$currPos); peg$currPos++; } else { s3 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c82); } + if (peg$silentFails === 0) { peg$fail(peg$c97); } } if (s3 === peg$FAILED) { s3 = null; } if (s3 !== peg$FAILED) { peg$savedPos = s0; - s1 = peg$c85(s2); + s1 = peg$c100(s2); s0 = s1; } else { peg$currPos = s0; @@ -1285,7 +1471,7 @@ module.exports = (function() { peg$silentFails--; if (s0 === peg$FAILED) { s1 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c80); } + if (peg$silentFails === 0) { peg$fail(peg$c95); } } return s0; @@ -1297,11 +1483,11 @@ module.exports = (function() { peg$silentFails++; s0 = peg$currPos; if (input.charCodeAt(peg$currPos) === 34) { - s1 = peg$c87; + s1 = peg$c102; peg$currPos++; } else { s1 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c88); } + if (peg$silentFails === 0) { peg$fail(peg$c103); } } if (s1 !== peg$FAILED) { s2 = []; @@ -1312,15 +1498,15 @@ module.exports = (function() { } if (s2 !== peg$FAILED) { if (input.charCodeAt(peg$currPos) === 34) { - s3 = peg$c87; + s3 = peg$c102; peg$currPos++; } else { s3 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c88); } + if (peg$silentFails === 0) { peg$fail(peg$c103); } } if (s3 !== peg$FAILED) { peg$savedPos = s0; - s1 = peg$c89(s2); + s1 = peg$c104(s2); s0 = s1; } else { peg$currPos = s0; @@ -1337,11 +1523,11 @@ module.exports = (function() { if (s0 === peg$FAILED) { s0 = peg$currPos; if (input.charCodeAt(peg$currPos) === 39) { - s1 = peg$c90; + s1 = peg$c105; peg$currPos++; } else { s1 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c91); } + if (peg$silentFails === 0) { peg$fail(peg$c106); } } if (s1 !== peg$FAILED) { s2 = []; @@ -1352,15 +1538,15 @@ module.exports = (function() { } if (s2 !== peg$FAILED) { if (input.charCodeAt(peg$currPos) === 39) { - s3 = peg$c90; + s3 = peg$c105; peg$currPos++; } else { s3 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c91); } + if (peg$silentFails === 0) { peg$fail(peg$c106); } } if (s3 !== peg$FAILED) { peg$savedPos = s0; - s1 = peg$c89(s2); + s1 = peg$c104(s2); s0 = s1; } else { peg$currPos = s0; @@ -1399,7 +1585,7 @@ module.exports = (function() { } if (s2 !== peg$FAILED) { peg$savedPos = s0; - s1 = peg$c89(s2); + s1 = peg$c104(s2); s0 = s1; } else { peg$currPos = s0; @@ -1414,7 +1600,7 @@ module.exports = (function() { peg$silentFails--; if (s0 === peg$FAILED) { s1 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c86); } + if (peg$silentFails === 0) { peg$fail(peg$c101); } } return s0; @@ -1426,12 +1612,12 @@ module.exports = (function() { s0 = peg$currPos; s1 = peg$currPos; peg$silentFails++; - if (peg$c92.test(input.charAt(peg$currPos))) { + if (peg$c107.test(input.charAt(peg$currPos))) { s2 = input.charAt(peg$currPos); peg$currPos++; } else { s2 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c93); } + if (peg$silentFails === 0) { peg$fail(peg$c108); } } peg$silentFails--; if (s2 === peg$FAILED) { @@ -1446,11 +1632,11 @@ module.exports = (function() { peg$currPos++; } else { s2 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c94); } + if (peg$silentFails === 0) { peg$fail(peg$c109); } } if (s2 !== peg$FAILED) { peg$savedPos = s0; - s1 = peg$c95(s2); + s1 = peg$c110(s2); s0 = s1; } else { peg$currPos = s0; @@ -1463,17 +1649,17 @@ module.exports = (function() { if (s0 === peg$FAILED) { s0 = peg$currPos; if (input.charCodeAt(peg$currPos) === 92) { - s1 = peg$c96; + s1 = peg$c111; peg$currPos++; } else { s1 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c97); } + if (peg$silentFails === 0) { peg$fail(peg$c112); } } if (s1 !== peg$FAILED) { s2 = peg$parseEscapeSequence(); if (s2 !== peg$FAILED) { peg$savedPos = s0; - s1 = peg$c95(s2); + s1 = peg$c110(s2); s0 = s1; } else { peg$currPos = s0; @@ -1494,12 +1680,12 @@ module.exports = (function() { s0 = peg$currPos; s1 = peg$currPos; peg$silentFails++; - if (peg$c98.test(input.charAt(peg$currPos))) { + if (peg$c113.test(input.charAt(peg$currPos))) { s2 = input.charAt(peg$currPos); peg$currPos++; } else { s2 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c99); } + if (peg$silentFails === 0) { peg$fail(peg$c114); } } peg$silentFails--; if (s2 === peg$FAILED) { @@ -1514,11 +1700,11 @@ module.exports = (function() { peg$currPos++; } else { s2 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c94); } + if (peg$silentFails === 0) { peg$fail(peg$c109); } } if (s2 !== peg$FAILED) { peg$savedPos = s0; - s1 = peg$c95(s2); + s1 = peg$c110(s2); s0 = s1; } else { peg$currPos = s0; @@ -1531,17 +1717,17 @@ module.exports = (function() { if (s0 === peg$FAILED) { s0 = peg$currPos; if (input.charCodeAt(peg$currPos) === 92) { - s1 = peg$c96; + s1 = peg$c111; peg$currPos++; } else { s1 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c97); } + if (peg$silentFails === 0) { peg$fail(peg$c112); } } if (s1 !== peg$FAILED) { s2 = peg$parseEscapeSequence(); if (s2 !== peg$FAILED) { peg$savedPos = s0; - s1 = peg$c95(s2); + s1 = peg$c110(s2); s0 = s1; } else { peg$currPos = s0; @@ -1576,11 +1762,11 @@ module.exports = (function() { peg$currPos++; } else { s2 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c94); } + if (peg$silentFails === 0) { peg$fail(peg$c109); } } if (s2 !== peg$FAILED) { peg$savedPos = s0; - s1 = peg$c95(s2); + s1 = peg$c110(s2); s0 = s1; } else { peg$currPos = s0; @@ -1597,53 +1783,53 @@ module.exports = (function() { function peg$parseEscapeSequence() { var s0, s1; - if (peg$c100.test(input.charAt(peg$currPos))) { + if (peg$c115.test(input.charAt(peg$currPos))) { s0 = input.charAt(peg$currPos); peg$currPos++; } else { s0 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c101); } + if (peg$silentFails === 0) { peg$fail(peg$c116); } } if (s0 === peg$FAILED) { s0 = peg$currPos; if (input.charCodeAt(peg$currPos) === 110) { - s1 = peg$c102; + s1 = peg$c117; peg$currPos++; } else { s1 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c103); } + if (peg$silentFails === 0) { peg$fail(peg$c118); } } if (s1 !== peg$FAILED) { peg$savedPos = s0; - s1 = peg$c104(); + s1 = peg$c119(); } s0 = s1; if (s0 === peg$FAILED) { s0 = peg$currPos; if (input.charCodeAt(peg$currPos) === 114) { - s1 = peg$c105; + s1 = peg$c120; peg$currPos++; } else { s1 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c106); } + if (peg$silentFails === 0) { peg$fail(peg$c121); } } if (s1 !== peg$FAILED) { peg$savedPos = s0; - s1 = peg$c107(); + s1 = peg$c122(); } s0 = s1; if (s0 === peg$FAILED) { s0 = peg$currPos; if (input.charCodeAt(peg$currPos) === 116) { - s1 = peg$c108; + s1 = peg$c123; peg$currPos++; } else { s1 = peg$FAILED; - if (peg$silentFails === 0) { peg$fail(peg$c109); } + if (peg$silentFails === 0) { peg$fail(peg$c124); } } if (s1 !== peg$FAILED) { peg$savedPos = s0; - s1 = peg$c110(); + s1 = peg$c125(); } s0 = s1; } @@ -1722,6 +1908,30 @@ module.exports = (function() { responseCodeFilter.desc = "resp. code is " + code; return responseCodeFilter; } + function body(regex){ + regex = new RegExp(regex, "i"); + function bodyFilter(flow){ + return True; + } + bodyFilter.desc = "body filters are not implemented yet, see https://github.com/mitmproxy/mitmweb/issues/10"; + return bodyFilter; + } + function requestBody(regex){ + regex = new RegExp(regex, "i"); + function requestBodyFilter(flow){ + return True; + } + requestBodyFilter.desc = "body filters are not implemented yet, see https://github.com/mitmproxy/mitmweb/issues/10"; + return requestBodyFilter; + } + function responseBody(regex){ + regex = new RegExp(regex, "i"); + function responseBodyFilter(flow){ + return True; + } + responseBodyFilter.desc = "body filters are not implemented yet, see https://github.com/mitmproxy/mitmweb/issues/10"; + return responseBodyFilter; + } function domain(regex){ regex = new RegExp(regex, "i"); function domainFilter(flow){ @@ -1730,6 +1940,16 @@ module.exports = (function() { domainFilter.desc = "domain matches " + regex; return domainFilter; } + function destination(regex){ + regex = new RegExp(regex, "i"); + function destinationFilter(flow){ + return (!!flow.server_conn.address) + && + regex.test(flow.server_conn.address.address[0] + ":" + flow.server_conn.address.address[1]); + } + destinationFilter.desc = "destination address matches " + regex; + return destinationFilter; + } function errorFilter(flow){ return !!flow.error; } @@ -1786,7 +2006,16 @@ module.exports = (function() { return !!flow.response; } responseFilter.desc = "has response"; - + function source(regex){ + regex = new RegExp(regex, "i"); + function sourceFilter(flow){ + return (!!flow.client_conn.address) + && + regex.test(flow.client_conn.address.address[0] + ":" + flow.client_conn.address.address[1]); + } + sourceFilter.desc = "source address matches " + regex; + return sourceFilter; + } function contentType(regex){ regex = new RegExp(regex, "i"); function contentTypeFilter(flow){ diff --git a/web/src/js/filt/filt.peg b/web/src/js/filt/filt.peg index f3235ccd..989bfdd3 100644 --- a/web/src/js/filt/filt.peg +++ b/web/src/js/filt/filt.peg @@ -69,6 +69,30 @@ function responseCode(code){ responseCodeFilter.desc = "resp. code is " + code; return responseCodeFilter; } +function body(regex){ + regex = new RegExp(regex, "i"); + function bodyFilter(flow){ + return True; + } + bodyFilter.desc = "body filters are not implemented yet, see https://github.com/mitmproxy/mitmweb/issues/10"; + return bodyFilter; +} +function requestBody(regex){ + regex = new RegExp(regex, "i"); + function requestBodyFilter(flow){ + return True; + } + requestBodyFilter.desc = "body filters are not implemented yet, see https://github.com/mitmproxy/mitmweb/issues/10"; + return requestBodyFilter; +} +function responseBody(regex){ + regex = new RegExp(regex, "i"); + function responseBodyFilter(flow){ + return True; + } + responseBodyFilter.desc = "body filters are not implemented yet, see https://github.com/mitmproxy/mitmweb/issues/10"; + return responseBodyFilter; +} function domain(regex){ regex = new RegExp(regex, "i"); function domainFilter(flow){ @@ -77,6 +101,16 @@ function domain(regex){ domainFilter.desc = "domain matches " + regex; return domainFilter; } +function destination(regex){ + regex = new RegExp(regex, "i"); + function destinationFilter(flow){ + return (!!flow.server_conn.address) + && + regex.test(flow.server_conn.address.address[0] + ":" + flow.server_conn.address.address[1]); + } + destinationFilter.desc = "destination address matches " + regex; + return destinationFilter; +} function errorFilter(flow){ return !!flow.error; } @@ -133,7 +167,16 @@ function responseFilter(flow){ return !!flow.response; } responseFilter.desc = "has response"; - +function source(regex){ + regex = new RegExp(regex, "i"); + function sourceFilter(flow){ + return (!!flow.client_conn.address) + && + regex.test(flow.client_conn.address.address[0] + ":" + flow.client_conn.address.address[1]); + } + sourceFilter.desc = "source address matches " + regex; + return sourceFilter; +} function contentType(regex){ regex = new RegExp(regex, "i"); function contentTypeFilter(flow){ @@ -205,36 +248,36 @@ BindingExpr { return binding(expr); } / Expr -Expr - = NullaryExpr - / UnaryExpr +/* All the filters except "~s" and "~src" are arranged in the ascending order as + given in the docs(http://docs.mitmproxy.org/en/latest/features/filters.html). + "~s" and "~src" are so arranged as "~s" caused problems in the evaluation of + "~src". */ -NullaryExpr - = BooleanLiteral +Expr + = "true" { return trueFilter; } + / "false" { return falseFilter; } / "~a" { return assetFilter; } + / "~b" ws+ s:StringLiteral { return body(s); } + / "~bq" ws+ s:StringLiteral { return requestBody(s); } + / "~bs" ws+ s:StringLiteral { return responseBody(s); } + / "~c" ws+ s:IntegerLiteral { return responseCode(s); } + / "~d" ws+ s:StringLiteral { return domain(s); } + / "~dst" ws+ s:StringLiteral { return destination(s); } / "~e" { return errorFilter; } + / "~h" ws+ s:StringLiteral { return header(s); } + / "~hq" ws+ s:StringLiteral { return requestHeader(s); } + / "~hs" ws+ s:StringLiteral { return responseHeader(s); } / "~http" { return httpFilter; } + / "~m" ws+ s:StringLiteral { return method(s); } / "~marked" { return markedFilter; } / "~q" { return noResponseFilter; } + / "~src" ws+ s:StringLiteral { return source(s); } / "~s" { return responseFilter; } + / "~t" ws+ s:StringLiteral { return contentType(s); } / "~tcp" { return tcpFilter; } - - -BooleanLiteral - = "true" { return trueFilter; } - / "false" { return falseFilter; } - -UnaryExpr - = "~c" ws+ s:IntegerLiteral { return responseCode(s); } - / "~d" ws+ s:StringLiteral { return domain(s); } - / "~h" ws+ s:StringLiteral { return header(s); } - / "~hq" ws+ s:StringLiteral { return requestHeader(s); } - / "~hs" ws+ s:StringLiteral { return responseHeader(s); } - / "~m" ws+ s:StringLiteral { return method(s); } - / "~t" ws+ s:StringLiteral { return contentType(s); } / "~tq" ws+ s:StringLiteral { return requestContentType(s); } / "~ts" ws+ s:StringLiteral { return responseContentType(s); } - / "~u" ws+ s:StringLiteral { return url(s); } + / "~u" ws+ s:StringLiteral { return url(s); } / s:StringLiteral { return url(s); } IntegerLiteral "integer" |
