aboutsummaryrefslogtreecommitdiffstats
path: root/cloud_mdir_sync
Commit message (Collapse)AuthorAgeFilesLines
* Gmail: Retry on 502 Bad GatewayJason Gunthorpe2020-06-221-0/pre { line-height: 125%; margin: 0; } td.linenos pre { color: #000000; background-color: #f0f0f0; padding: 0 5px 0 5px; } span.linenos { color: #000000; background-color: #f0f0f0; padding: 0 5px 0 5px; } td.linenos pre.special { color: #000000; background-color: #ffffc0; padding: 0 5px 0 5px; } span.linenos.special { color: #000000; background-color: #ffffc0; padding: 0 5px 0 5px; } .highlight .hll { background-color: #ffffcc } .highlight { background: #ffffff; } .highlight .c { color: #888888 } /* Comment */ .highlight .err { color: #a61717; background-color: #e3d2d2 } /* Error */ .highlight .k { color: #008800; font-weight: bold } /* Keyword */ .highlight .ch { color: #888888 } /* Comment.Hashbang */ .highlight .cm { color: #888888 } /* Comment.Multiline */ .highlight .cp { color: #cc0000; font-weight: bold } /* Comment.Preproc */ .highlight .cpf { color: #888888 } /* Comment.PreprocFile */ .highlight .c1 { color: #888888 } /* Comment.Single */ .highlight .cs { color: #cc0000; font-weight: bold; background-color: #fff0f0 } /* Comment.Special */ .highlight .gd { color: #000000; background-color: #ffdddd } /* Generic.Deleted */ .highlight .ge { font-style: italic } /* Generic.Emph */ .highlight .gr { color: #aa0000 } /* Generic.Error */ .highlight .gh { color: #333333 } /* Generic.Heading */ .highlight .gi { color: #000000; background-color: #ddffdd } /* Generic.Inserted */ .highlight .go { color: #888888 } /* Generic.Output */ .highlight .gp { color: #555555 } /* Generic.Prompt */ .highlight .gs { font-weight: bold } /* Generic.Strong */ .highlight .gu { color: #666666 } /* Generic.Subheading */ .highlight .gt { color: #aa0000 } /* Generic.Traceback */ .highlight .kc { color: #008800; font-weight: bold } /* Keyword.Constant */ .highlight .kd { color: #008800; font-weight: bold } /* Keyword.Declaration */ .highlight .kn { color: #008800; font-weight: bold } /* Keyword.Namespace */ .highlight .kp { color: #008800 } /* Keyword.Pseudo */ .highlight .kr { color: #008800; font-weight: bold } /* Keyword.Reserved */ .highlight .kt { color: #888888; font-weight: bold } /* Keyword.Type */ .highlight .m { color: #0000DD; font-weight: bold } /* Literal.Number */ .highlight .s { color: #dd2200; background-color: #fff0f0 } /* Literal.String */ .highlight .na { color: #336699 } /* Name.Attribute */ .highlight .nb { color: #003388 } /* Name.Builtin */ .highlight .nc { color: #bb0066; font-weight: bold } /* Name.Class */ .highlight .no { color: #003366; font-weight: bold } /* Name.Constant */ .highlight .nd { color: #555555 } /* Name.Decorator */ .highlight .ne { color: #bb0066; font-weight: bold } /* Name.Exception */ .highlight .nf { color: #0066bb; font-weight: bold } /* Name.Function */ .highlight .nl { color: #336699; font-style: italic } /* Name.Label */ .highlight .nn { color: #bb0066; font-weight: bold } /* Name.Namespace */ .highlight .py { color: #336699; font-weight: bold } /* Name.Property */ .highlight .nt { color: #bb0066; font-weight: bold } /* Name.Tag */ .highlight .nv { color: #336699 } /* Name.Variable */ .highlight .ow { color: #008800 } /* Operator.Word */ .highlight .w { color: #bbbbbb } /* Text.Whitespace */ .highlight .mb { color: #0000DD; font-weight: bold } /* Literal.Number.Bin */ .highlight .mf { color: #0000DD; font-weight: bold } /* Literal.Number.Float */ .highlight .mh { color: #0000DD; font-weight: bold } /* Literal.Number.Hex */ .highlight .mi { color: #0000DD; font-weight: bold } /* Literal.Number.Integer */ .highlight .mo { color: #0000DD; font-weight: bold } /* Literal.Number.Oct */ .highlight .sa { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Affix */ .highlight .sb { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Backtick */ .highlight .sc { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Char */ .highlight .dl { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Delimiter */ .highlight .sd { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Doc */ .highlight .s2 { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Double */ .highlight .se { color: #0044dd; background-color: #fff0f0 } /* Literal.String.Escape */ .highlight .sh { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Heredoc */ .highlight .si { color: #3333bb; background-color: #fff0f0 } /* Literal.String.Interpol */ .highlight .sx { color: #22bb22; background-color: #f0fff0 } /* Literal.String.Other */ .highlight .sr { color: #008800; background-color: #fff0ff } /* Literal.String.Regex */ .highlight .s1 { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Single */ .highlight .ss { color: #aa6600; background-color: #fff0f0 } /* Literal.String.Symbol */ .highlight .bp { color: #003388 } /* Name.Builtin.Pseudo */ .highlight .fm { color: #0066bb; font-weight: bold } /* Name.Function.Magic */ .highlight .vc { color: #336699 } /* Name.Variable.Class */ .highlight .vg { color: #dd7700 } /* Name.Variable.Global */ .highlight .vi { color: #3333bb } /* Name.Variable.Instance */ .highlight .vm { color: #336699 } /* Name.Variable.Magic */ .highlight .il { color: #0000DD; font-weight: bold } /* Literal.Number.Integer.Long */
def request(flow):
    if flow.request.urlencoded_form:
        # If there's already a form, one can just add items to the dict:
        flow.request.urlencoded_form["mitmproxy"] = "rocks"
    else:
        # One can also just pass new form data.
        # This sets the proper content type and overrides the body.
        flow.request.urlencoded_form = [
            ("foo", "bar")
        ]
2-0/+8 | | | | | | | | MailDir writes cause the inode mtime and ctime to change which causes the cache to become invalid. Update the mtime and ctime information after changing it. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com> * Speed up loading large databases on startupJason Gunthorpe2020-05-301-14/+21 | | | | | | Batching the sha1sum work is dramatically faster. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com> * Increase the page size for listing O365 mailboxesJason Gunthorpe2020-05-301-0/+1 | | | | | | | This helps large mailboxes quite a lot, it looks like this can list some 700 messages/second Signed-off-by: Jason Gunthorpe <jgg@mellanox.com> * Manage concurrency to avoid running out of file descriptorsJason Gunthorpe2020-05-302-29/+33 | | | | | | | | | | | | AIO does not seem to strictly pipeline work, so it is possible for a very large number of message download tasks to be scheduled. Each one that gets scheduled opens a file descriptor, and so we can run out of them fairly fast. Strictly limit the number of open files per mailbox during message downloading. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com> * Add cms-oauthJason Gunthorpe2020-05-281-0/+84 | | | | | | | | | | | This is a command line program to get the OAUTH tokens from the credential server. It is intended to fit into the 'call a program to get the token' methodology that several tools are implementing. Several options are provided to format the token and a built in SMTP protocol tests that the server is working properly. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com> * Add OAUTH Credential serverJason Gunthorpe2020-05-285-15/+143 | | | | | | | | | | | | | | | | The OAUTH credential server allows CMS to ack as an OAUTH broker and supply bearer tokens to other applications in the system. Currently this only support SMTP tokens for outbound mail delivery. A UNIX domain socket is used to communicate between the SMTP agent and CMS. A simple one line protocol is used to specify the account requested and CMS returns the plain XAOUTH2 response string. The agent is responsible to base64 encode it. This works for GMail and O365 mailboxes. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com> * Remove cfg.domainsJason Gunthorpe2020-05-284-55/+42 | | | | | | | | Directly connect the 'account' API objects to their mailbox users through the config language instead of trying to fix it up after the fact with a dictionary. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com> * o365: Support delete_action in O365Jason Gunthorpe2020-03-271-1/+6 | | | | | | | If set to "delete" the deleted messages will go to Deleted Items by default things go to Archive, just like gmail. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com> * gmail: Fix determining the start_history_idJason Gunthorpe2020-02-241-7/+13 | | | | | | | | | | | | Occasionally gmail returns history records that have junk in them, for instance messages IDs that don't exist. This appears to be a bug. Using the latest history ID seems to guarantee that a subsequent history query will be empty and we can then avoid the junk and start again. Go back to full query if delta fails for any reason. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com> * Catch failures from the sha1sum forkJason Gunthorpe2020-02-211-1/+2 | | | | | | | | This should trigger a sleep and retry, not a program failure. It happens if the sha1sum races with the user renaming a MailDir file. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com> * Fix an exception if message fetching failsJason Gunthorpe2020-02-212-0/+2 | | | | | | | Since size is used to print the log message, even on failure, set it to some value. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com> * Don't leave asyncio tasks running unexpectedlyJason Gunthorpe2020-02-074-13/+29 | | | | | | | | All cases where gather is called intend that the tasks will complete successfully or all cancel at the first error. Add a little wrapper to achieve this. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com> * Allow access to the message headers during mail directingJason Gunthorpe2020-02-072-30/+48 | | | | | | | Access to the headers like List-ID allows some client side filtering of incoming messages. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com> * Do not pass msgdb all over the placeJason Gunthorpe2020-02-075-39/+38 | | | | | | | The mailboxes can only be linked to a single msgdb, always take it from the mailbox cfg. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com> * Lift Mailbox.cfg out of the sub classesJason Gunthorpe2020-02-076-20/+22 | | | | | | Make sure it is always set Signed-off-by: Jason Gunthorpe <jgg@mellanox.com> * GMail supportJason Gunthorpe2020-02-065-11/+671 | | | | | | Basic support for GMail using the REST API Signed-off-by: Jason Gunthorpe <jgg@mellanox.com> * Allow a Mailbox to search for messages with a None email_idJason Gunthorpe2020-02-051-16/+40 | | | | | | | Gmail does not have easy access to the email_id, but does have a stable storage_id, so provide a second dictionary. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com> * Preserve flags in the local MailDir that are not supported in the cloudJason Gunthorpe2020-02-052-7/+24 | | | | | | This is needed for gmail, since it doesn't support all of the flags. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com> * Do not run tracing by defaultJason Gunthorpe2020-02-052-4/+2 | | | | | | trace is only for debugging adventures. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com> * Add some more type annotations to O365MailboxJason Gunthorpe2020-02-042-10/+14 | | | | Signed-off-by: Jason Gunthorpe <jgg@mellanox.com> * Add a --offline optionJason Gunthorpe2020-01-311-10/+32 | | | | | | | | | | | This allows treating the local message flags as authoritative during program startup. Messages marked locally as deleted will be deleted on the server during startup. For safety, this never considers the absence of a message locally as a deletion. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com> * Support FLAG_DELETED in Office365Jason Gunthorpe2020-01-311-16/+26 | | | | | | | | | | | | | | When a local message is tagged deleted then update the cloud flags and delete the message. Using the Trash flag in a mail dir allows the cloud upload step to see the original message and then upload the local flags before deleting it. For instance, this will allow the Replied flag to be set upon deletion. Once deleted in the cloud the next sync iteration will delete the message locally. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com> * Support the Trash flag in MailDirJason Gunthorpe2020-01-312-3/+8 | | | | | | | This can be set by mutt when 'set maildir_trash=yes' and shows as a deleted item. Keep track of it on the maildir side. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com> * Move Office365 flag manipulation into a functionJason Gunthorpe2020-01-311-67/+70 | | | | | | For reuse in the next patches Signed-off-by: Jason Gunthorpe <jgg@mellanox.com> * Trigger an update of a O365 mailbox if the notification FD closesJason Gunthorpe2020-01-181-0/+2 | | | | | | Fixes an issue where mailbox changes could be missed. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com> * Initial commit of cloud_mdir_syncJason Gunthorpe2020-01-109-0/+1607 I have been using for a few months now with no ill effects. Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>