| Age | Commit message (Collapse) | Author |
|---|
|
- cleanup the internal queue backend API and get rid of the QOP_* thing.
- implement a queue_proc backend
- rename queue_fsqueue.c to queue_fs
- enable support for queue encryption
- add an envelope cache
- better logging and error reporting
|
|
now happens in queue_init(), and backends take the queue passwd as
parameter in their init function.
Remove useless SMTPD_FILTER_USER while there.
|
|
ok gilles@
|
|
ok gilles
|
|
* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright
FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.
smtpd.conf(5) simplified, it will require adaptations
ok eric@
|
|
a simple QOP_WALK queue operation. Some knf and formating fixes while
there.
ok gilles@
|
|
* Log more events (especially client session) and use a better scheme
for that: each messages is prefixed with a token to easily identify
its class:
- info/warn/debug: general server messages
- smtp-in: smtp client connections
- relay: status update for relayed messages
- delivery: status update for local deliveries
* Implement "smtpctl monitor" to display updates of selected internal
counters.
* When reloading the on-disk queue at startup do not commit a message
if no envelope was submitted for that message.
* Remove unused stuff in the config parser.
ok gilles@
|
|
ok gilles@
|
|
- remove support for encrypted queue, it will be reintroduced later after
pouring more thinking into it
if you had it enabled, flush your queue before updating
|
|
crypto_backend
ok gilles@
|
|
ok gilles@
|
|
using the new API. By default, OpenSMTPD does not provide queue encryption,
but it can be enabled with "queue encryption [args]" and will transparently
encrypt/decrypt envelopes/messages as they hit the queue.
By default, it will use Blowfish in CBC mode with a different random IV for
each envelope and message. User provided key is expanded using sha256 but a
different cipher and digest may be specified in smtpd.conf
Queue encryption is compatible with compression and if both options are set
it will do them in correct order and transparently.
tested by chl@, a few users and myself
ok chl@ and I
|
|
- comment variables unused at this time
|
|
queue.
To use it, just add "queue compress" in smtpd.conf. For now, only zlib is used.
lots of feedback from eric@ and gilles@
ok eric@ gilles@
|
|
|
|
- remove loading of evpid.
- don't dump the msgid
- ignore msgid at load
- remove now unused functions ascii_{dump,load}_uint{32,64}_hex()
With inputs from eric@ and gilles@
ok gilles@ eric@
|
|
envelope id and
an envelope ascii buffer.
ok eric@ gilles@
|
|
ok eric@
|
|
- remove the /envelopes subdirectory, envelopes are at the same level than
the message file
- kill PATH_ENVELOPES define
but keep the:
- reduce the number of buckets from 0xfff to 0xff, this avoid performances
of the queue to decrease when we start having tons of buckets
ok eric@ gilles@
|
|
- remove the /envelopes subdirectory, envelopes are at the same level than
the message file
- kill PATH_ENVELOPES define
- reduce the number of buckets from 0xfff to 0xff, this avoid performances
of the queue to decrease when we start having tons of buckets
this diff introduces a change to the queue layout, you will want to empty
your queue before updating. more cleanup to come
ok eric@, ok chl@
|
|
incoming messages are now always stored in /incoming, whatever the queue_backend is.
remove QOP_FD_RW and fsqueue_message_fd_rw().
while there check return value of generated paths before calling rmtree()
with advice from gilles@ and eric@
ok gilles@ eric@
|
|
Add a command-line option to specify the backend to use at runtime.
ok gilles@
|
|
internally in fsqueue backend for now, and let the fsqueue_message()
and fsqueue_envelope() dispatchers do the right thing.
Based on a diff by chl@
ok chl@ gilles@
|
|
ok eric@ gilles@
|
|
Make sure existing envelopes can be properly loaded.
ok chl@ gilles@
|
|
the wrapper function rather than in individual backends.
ok gilles@
|
|
ok gilles@ eric@
|
|
thing. Remove queue_message_purge() in favor of queue_message_delete
and simplify fsqueue_message_delete() implementation to move the
message dir to purge/
ok gilles@
|
|
current confusing hack.
ok gilles@
|
|
- return an informative error string if the envelope is invalid.
- take the envelope id as a parameter and make sure it matches.
- do not expect the errorline to start with an SMTP response code,
as this is not always the case: a temporary failure with mda would
cause the envelope to be marked as corrupted. Instead, just make sure
that all string fields are actual strings to prevent overflows later.
ok gilles@ chl@
|
|
ok chl@ gilles@
|
|
ok gilles@
|
|
queue_backend into account and assumed a filesystem with a specific layout.
This commit does plenty of things:
- make qwalk an abstraction in the queue_backend API, and impose queue
drivers to implement qwalk_open(), qwalk() and qwalk_close();
- move previous qwalk_open(), qwalk() and qwalk_close() to the fsqueue
driver since they were fsqueue specific ...
- make qwalk API work with msgid/evpid instead of pathnames since we're
going to use the queue_backend API to load envelopes by evpid anyway;
- makes smtpd use *solely* the queue_backend API when manipulating the
queue. pathnames were removed from smtpd.h and moved into the fsqueue
which means we can now store a queue anywhere ... as long as we write
the ten functions or so required for a queue driver ;-)
ok eric@, ok chl@
|
|
schedule queue to corrupt queue upon envelope loading failure.
tested by me, ok eric@
|
|
couple of load/dump functions to convert to and from a human readable fmt.
while at it kill struct delivery and merge back its fields to the envelope.
this basically means we shouldn't require users to flush their queues every
time we make a change to struct envelope.
work is not done, but we're at a better state than the binary fsqueue so
we'll improve it in-tree.
has been running on my own box for the last 12 hours or so
ok eric@, chl@
|
|
bring a shitload of unnecessary information everywhere. this required many
parts of smtpd to be refactored and more specifically envelope expansion.
in the process lots of code got simplified, and the envelope expansion code
has been isolated to lka_session.c with some longstanding bugs fixed.
Diff has been tested by many with no major regression reported.
armani@ spotted a bug in a setup where a domain is listed a both primary
and virtual, I will fix that in-tree as it's becoming painful to maintain
this diff out.
|
|
discussed with and ok gilles@
|
|
struct message ...
|
|
smtpd now has an evpid associated to each delivery message, the evpid is an
u_int64_t where the upper 32 bits are the msgid, and the 32 bits are the
envelope unique identifier for that message. this results in lots of space
saved in both disk-based and ram-based queues, but also simplifies a lot of
code.
change has been stressed on my desktop, and has ran on my MX for the entire
afternoon without a regression.
|
|
fsqueue_envelope_create(), fsqueue_message_purge()
- kill deprecated functions in queue_shared.c
At this point fsqueue backend is almost complete, all that is left to do is
to move the qwalk() API inside the queue_backend API, then make sure smtpd
is no longer calling anything queue related directly.
|
|
- fsqueue->setup() performs the queue initialization;
- fsqueue->message() controls messages;
- fsqueue->envelope() controls envelopes;
This commit brings the following to fsbackend:
fsqueue_setup(), fsqueue_message_delete(), fsqueue_envelope_load(),
fsqueue_envelope_update(), fsqueue_envelope_delete().
It also makes smtpd use the queue_backend API for these operations.
|
|
disk-based queue, it makes it near impossible to make changes to it without
editing twenty files... how am i going to implement mongodb support ? :-)
bring a new queue_backend API which hides the details of the disk-based
queue to smtpd. it is not "plugged in" yet and I'm filling the holes.
|
|
be done before we can write queue backends anyway ...
|
|
|
|
|
|
Major goals:
1) Fix bad performance caused by the runner process doing full queue
read in 1s intervals. My Soekris can now happily accept >50 msg/s
while having multi-thousand queue; before, one hundred queue would
bring the system to its knees.
2) Introduce Qmail-like scheduler that doesn't write as much to the
disk so that it needs less code for servicing error conditions,
which in some places can be tricky to get right.
3) Introduce separation between the scheduler and the backend; these
two queue aspects shouldn't be too tied too each other. This means
that eg. storing queue in SQL requires rewrite of just queue_backend.c.
4) Make on-disk queue format architecture independent, and more
easily extensible, to reduce number of flag days in the future.
Minor goals:
ENOSPC no longer prevents delivery attempts, fixed session limiting
for relayed mail, improved batching of "relay via" mails, human-readable
mailq output, "show queue raw" command, clearer logging, sending
of single bounce about multiple recipients, exact delay= computation,
zero delay between deliveries while within session limit (currently
1s delay between re-scheduling is enforced), mta no longer requests
content fd, corrected session limit for bounce submissions, tiny
<100B queue files instead of multi-KB, detect loops before accepting
mail, reduce traffic on imsg channels by killing enormous struct
submit_status.
|
