From fdb6a44245249a50b5c95cdf0d8d13ecddfe5726 Mon Sep 17 00:00:00 2001 From: Aldo Cortesi Date: Wed, 12 Oct 2016 10:57:05 +1300 Subject: docs: cleanups improvements and fighting sphinx - Hide links to internal code listings, and link to github instead - Improve formatting of code/example captions - Fix outdated documentation of command-line options - Complete documentation of all events + improved formatting - tcp_open -> tcp_start, tcp_close -> tcp_end to reduce confusion --- docs/scripting/overview.rst | 79 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+) create mode 100644 docs/scripting/overview.rst (limited to 'docs/scripting/overview.rst') diff --git a/docs/scripting/overview.rst b/docs/scripting/overview.rst new file mode 100644 index 00000000..a0dfe111 --- /dev/null +++ b/docs/scripting/overview.rst @@ -0,0 +1,79 @@ +.. _overview: + +Overview +========= + +Mitmproxy has a powerful scripting API that allows you to control almost any +aspect of traffic being proxied. In fact, much of mitmproxy's own core +functionality is implemented using the exact same API exposed to scripters (see +:src:`mitmproxy/builtins`). + +Scripting is event driven, with named handlers on the script object called at +appropriate points of mitmproxy's operation. Here's a complete mitmproxy script +that adds a new header to every HTTP response before it is returned to the +client: + +.. literalinclude:: ../../examples/add_header.py + :caption: :src:`examples/add_header.py` + :language: python + +All events that deal with an HTTP request get an instance of +:py:class:`~mitmproxy.models.HTTPFlow`, which we can use to manipulate the +response itself. We can now run this script using mitmdump or mitmproxy as +follows: + +>>> mitmdump -s add_header.py + +The new header will be added to all responses passing through the proxy. + + +mitmproxy comes with a variety of example inline scripts, which demonstrate +many basic tasks. + + +Running scripts in parallel +--------------------------- + +We have a single flow primitive, so when a script is blocking, other requests are not processed. +While that's usually a very desirable behaviour, blocking scripts can be run threaded by using the +:py:obj:`mitmproxy.script.concurrent` decorator. +**If your script does not block, you should avoid the overhead of the decorator.** + +.. literalinclude:: ../../examples/nonblocking.py + :caption: examples/nonblocking.py + :language: python + +Make scripts configurable with arguments +---------------------------------------- + +Sometimes, you want to pass runtime arguments to the inline script. This can be simply done by +surrounding the script call with quotes, e.g. ```mitmdump -s 'script.py --foo 42'``. +The arguments are then exposed in the start event: + +.. literalinclude:: ../../examples/modify_response_body.py + :caption: examples/modify_response_body.py + :language: python + + +Running scripts on saved flows +------------------------------ + +Sometimes, we want to run a script on :py:class:`~mitmproxy.models.Flow` objects that are already +complete. This happens when you start a script, and then load a saved set of flows from a file +(see the "scripted data transformation" example :ref:`here `). +It also happens when you run a one-shot script on a single flow through the ``|`` (pipe) shortcut +in mitmproxy. + +In this case, there are no client connections, and the events are run in the following order: +**start**, **request**, **responseheaders**, **response**, **error**, **done**. +If the flow doesn't have a **response** or **error** associated with it, the matching events will +be skipped. + +Spaces in the script path +------------------------- + +By default, spaces are interpreted as a separator between the inline script and its arguments +(e.g. ``-s 'foo.py 42'``). Consequently, the script path needs to be wrapped in a separate pair of +quotes if it contains spaces: ``-s '\'./foo bar/baz.py\' 42'``. + +.. _GitHub: https://github.com/mitmproxy/mitmproxy -- cgit v1.2.3 From fb69c9c3453142ae6beb4040295accb41fdc6878 Mon Sep 17 00:00:00 2001 From: Aldo Cortesi Date: Sun, 16 Oct 2016 11:12:58 +1300 Subject: docs: overview, classes, arguments --- docs/scripting/overview.rst | 72 ++++++++++++++++++++++++++++++++------------- 1 file changed, 52 insertions(+), 20 deletions(-) (limited to 'docs/scripting/overview.rst') diff --git a/docs/scripting/overview.rst b/docs/scripting/overview.rst index a0dfe111..f8dd9f2e 100644 --- a/docs/scripting/overview.rst +++ b/docs/scripting/overview.rst @@ -1,13 +1,17 @@ .. _overview: -Overview -========= +Introduction +============ Mitmproxy has a powerful scripting API that allows you to control almost any aspect of traffic being proxied. In fact, much of mitmproxy's own core functionality is implemented using the exact same API exposed to scripters (see :src:`mitmproxy/builtins`). + +A simple example +---------------- + Scripting is event driven, with named handlers on the script object called at appropriate points of mitmproxy's operation. Here's a complete mitmproxy script that adds a new header to every HTTP response before it is returned to the @@ -17,18 +21,57 @@ client: :caption: :src:`examples/add_header.py` :language: python -All events that deal with an HTTP request get an instance of -:py:class:`~mitmproxy.models.HTTPFlow`, which we can use to manipulate the -response itself. We can now run this script using mitmdump or mitmproxy as -follows: +All events that deal with an HTTP request get an instance of `HTTPFlow +`_, which we can use to manipulate the +response itself. We can now run this script using mitmdump, and the new header +will be added to all responses passing through the proxy: >>> mitmdump -s add_header.py -The new header will be added to all responses passing through the proxy. +Using classes +------------- + +In the example above, the script object is the ``add_header`` module itself. +That is, the handlers are declared at the global level of the script. This is +great for quick hacks, but soon becomes limiting as scripts become more +sophisticated. + +When a script first starts up, the `start `_, event is +called before anything else happens. You can replace the current script object +by returning it from this handler. Here's how this looks when applied to the +example above: + +.. literalinclude:: ../../examples/classes.py + :caption: :src:`examples/classes.py` + :language: python + +So here, we're using a module-level script to "boot up" into a class instance. +From this point on, the module-level script is removed from the handler chain, +and is replaced by the class instance. + + +Handling arguments +------------------ -mitmproxy comes with a variety of example inline scripts, which demonstrate -many basic tasks. +Scripts can handle their own command-line arguments, just like any other Python +program. Let's build on the example above to do something slightly more +sophisticated - replace one value with another in all responses. Mitmproxy's +`HTTPRequest `_ and `HTTPResponse +`_ objects have a handy `replace +`_ method that takes care +of all the details for us. + +.. literalinclude:: ../../examples/arguments.py + :caption: :src:`examples/arguments.py` + :language: python + +We can now call this script on the command-line like this: + +>>> mitmdump -dd -s "./arguments.py html faketml" + +Whenever a handler is called, mitpmroxy rewrites the script environment so that +it sees its own arguments as if it was invoked from the command-line. Running scripts in parallel @@ -43,17 +86,6 @@ While that's usually a very desirable behaviour, blocking scripts can be run thr :caption: examples/nonblocking.py :language: python -Make scripts configurable with arguments ----------------------------------------- - -Sometimes, you want to pass runtime arguments to the inline script. This can be simply done by -surrounding the script call with quotes, e.g. ```mitmdump -s 'script.py --foo 42'``. -The arguments are then exposed in the start event: - -.. literalinclude:: ../../examples/modify_response_body.py - :caption: examples/modify_response_body.py - :language: python - Running scripts on saved flows ------------------------------ -- cgit v1.2.3 From 55cb2a85472de8698b3dabc7ddc920b930e355d9 Mon Sep 17 00:00:00 2001 From: Aldo Cortesi Date: Sun, 16 Oct 2016 12:03:57 +1300 Subject: docs: logging and the context --- docs/scripting/overview.rst | 43 +++++++++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 14 deletions(-) (limited to 'docs/scripting/overview.rst') diff --git a/docs/scripting/overview.rst b/docs/scripting/overview.rst index f8dd9f2e..a3b83e44 100644 --- a/docs/scripting/overview.rst +++ b/docs/scripting/overview.rst @@ -74,18 +74,24 @@ Whenever a handler is called, mitpmroxy rewrites the script environment so that it sees its own arguments as if it was invoked from the command-line. -Running scripts in parallel ---------------------------- +Logging and the context +----------------------- -We have a single flow primitive, so when a script is blocking, other requests are not processed. -While that's usually a very desirable behaviour, blocking scripts can be run threaded by using the -:py:obj:`mitmproxy.script.concurrent` decorator. -**If your script does not block, you should avoid the overhead of the decorator.** +Scripts should not output straight to stderr or stdout. Instead, the `log +`_ object on the ``ctx`` contexzt module +should be used, so that the mitmproxy host program can handle output +appropriately. So, mitmdump can print colorised sript output to the terminal, +and mitmproxy console can place script output in the event buffer. -.. literalinclude:: ../../examples/nonblocking.py - :caption: examples/nonblocking.py +Here's how this looks: + +.. literalinclude:: ../../examples/logging.py + :caption: :src:`examples/logging.py` :language: python +The ``ctx`` module also exposes the mitmproxy master object at ``ctx.master`` +for advanced usage. + Running scripts on saved flows ------------------------------ @@ -101,11 +107,20 @@ In this case, there are no client connections, and the events are run in the fol If the flow doesn't have a **response** or **error** associated with it, the matching events will be skipped. -Spaces in the script path -------------------------- -By default, spaces are interpreted as a separator between the inline script and its arguments -(e.g. ``-s 'foo.py 42'``). Consequently, the script path needs to be wrapped in a separate pair of -quotes if it contains spaces: ``-s '\'./foo bar/baz.py\' 42'``. +Concurrency +----------- + +We have a single flow primitive, so when a script is blocking, other requests +are not processed. While that's usually a very desirable behaviour, blocking +scripts can be run threaded by using the :py:obj:`mitmproxy.script.concurrent` +decorator. + +.. literalinclude:: ../../examples/nonblocking.py + :caption: :src:`examples/nonblocking.py` + :language: python + -.. _GitHub: https://github.com/mitmproxy/mitmproxy + +Developing scripts +------------------ -- cgit v1.2.3 From 57b8ed21a9a30eb79d9340d5e146e42bbafd0d46 Mon Sep 17 00:00:00 2001 From: Aldo Cortesi Date: Sun, 16 Oct 2016 18:25:59 +1300 Subject: docs: scripts on saved flows --- docs/scripting/overview.rst | 30 +++++++++++++++++++++++------- 1 file changed, 23 insertions(+), 7 deletions(-) (limited to 'docs/scripting/overview.rst') diff --git a/docs/scripting/overview.rst b/docs/scripting/overview.rst index a3b83e44..744f5eb4 100644 --- a/docs/scripting/overview.rst +++ b/docs/scripting/overview.rst @@ -1,7 +1,7 @@ .. _overview: -Introduction -============ +Overview +======== Mitmproxy has a powerful scripting API that allows you to control almost any aspect of traffic being proxied. In fact, much of mitmproxy's own core @@ -96,11 +96,27 @@ for advanced usage. Running scripts on saved flows ------------------------------ -Sometimes, we want to run a script on :py:class:`~mitmproxy.models.Flow` objects that are already -complete. This happens when you start a script, and then load a saved set of flows from a file -(see the "scripted data transformation" example :ref:`here `). -It also happens when you run a one-shot script on a single flow through the ``|`` (pipe) shortcut -in mitmproxy. +When a flow is loaded from disk, the sequence of events that the flow would +have gone through on the wire is partially replayed. So, for instance, an HTTP +flow loaded from disk will trigger `requestheaders +`_, `request `_, +`responseheaders `_ and `response +`_ in order. We can use this behaviour to transform saved +traffic using scripts. For example, we can invoke the replacer script from +above on saved traffic as follows: + +>>> mitmdump -dd -s "./arguments.py html faketml" + + + + + +:py:class:`~mitmproxy.models.Flow` +objects that are already complete. This happens when you start a script, and +then load a saved set of flows from a file (see the "scripted data +transformation" example :ref:`here `). It also happens when you run a +one-shot script on a single flow through the ``|`` (pipe) shortcut in +mitmproxy. In this case, there are no client connections, and the events are run in the following order: **start**, **request**, **responseheaders**, **response**, **error**, **done**. -- cgit v1.2.3 From 00603021d9d486e3e16511eee273d26f59a3ab10 Mon Sep 17 00:00:00 2001 From: Aldo Cortesi Date: Sun, 16 Oct 2016 20:21:43 +1300 Subject: docs: concurrency, developing scripts --- docs/scripting/overview.rst | 39 +++++++++++++++++++-------------------- 1 file changed, 19 insertions(+), 20 deletions(-) (limited to 'docs/scripting/overview.rst') diff --git a/docs/scripting/overview.rst b/docs/scripting/overview.rst index 744f5eb4..5966eb1d 100644 --- a/docs/scripting/overview.rst +++ b/docs/scripting/overview.rst @@ -105,38 +105,37 @@ flow loaded from disk will trigger `requestheaders traffic using scripts. For example, we can invoke the replacer script from above on saved traffic as follows: ->>> mitmdump -dd -s "./arguments.py html faketml" - - - +>>> mitmdump -dd -s "./arguments.py html fakehtml" -r saved -w changed +This command starts the ``arguments`` script, reads all the flows from +``saved`` transforming them in the process, then writes them all to +``changed``. -:py:class:`~mitmproxy.models.Flow` -objects that are already complete. This happens when you start a script, and -then load a saved set of flows from a file (see the "scripted data -transformation" example :ref:`here `). It also happens when you run a -one-shot script on a single flow through the ``|`` (pipe) shortcut in -mitmproxy. - -In this case, there are no client connections, and the events are run in the following order: -**start**, **request**, **responseheaders**, **response**, **error**, **done**. -If the flow doesn't have a **response** or **error** associated with it, the matching events will -be skipped. +The mitmproxy console tool provides interactive ways to run transforming +scripts on flows - for instance, you can run a one-shot script on a single flow +through the ``|`` (pipe) shortcut. Concurrency ----------- -We have a single flow primitive, so when a script is blocking, other requests -are not processed. While that's usually a very desirable behaviour, blocking -scripts can be run threaded by using the :py:obj:`mitmproxy.script.concurrent` -decorator. +The mitmproxy script mechanism is single threaded, and the proxy blocks while +script handlers execute. This hugely simplifies the most common case, where +handlers are light-weight and the blocking doesn't have a performance impact. +It's possible to implement a concurrent mechanism on top of the blocking +framework, and mitmproxy includes a handy example of this that is fit for most +purposes. You can use it as follows: .. literalinclude:: ../../examples/nonblocking.py :caption: :src:`examples/nonblocking.py` :language: python - Developing scripts ------------------ + +Mitmprxoy monitors scripts for modifications, and reloads them on change. When +this happens, the script is shut down (the `done `_ event is +called), and the new instance is started up as if the script had just been +loaded (the `start `_ and `configure +`_ events are called). -- cgit v1.2.3