wilderjds
/
kmail
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Move the contents of our DESIGN file into a newly created Mainpage.dox

Browse Source 
and start doxygening them, such that they are more accessible and we
have better index page for the kmail apidox. With the automatic cross
linking this file becomes much more useful.

svn path=/trunk/KDE/kdepim/; revision=428782
wilder-work
Till Adam 21 years ago
parent 4e28cefc4a
commit 786166bb26
2 changed files with 870 additions and 872 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Split View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 873
      DESIGN
  2. 869
      Mainpage.dox

873
DESIGN
Unescape View File

@ -1,872 +1 @@
KMail design principles
=======================
This file is intended to guide the reader's way through the KMail
codebase. It should esp. be handy for people not hacking full-time on
KMail as well as people that want to trace bugs in parts of KMail
which they don't know well.
Contents:
- Kernel
- Identity
- Filters
- ConfigureDialog
- MDNs
- Folders
- Index
- Headers
- Display
TODO: reader, composer, messages, accounts, ...
KERNEL
======
Files: kmkernel.{h,cpp}
Contact Zack Rusin <zack@kde.org> with questions...
The first thing you'll notice about KMail is the extensive use of
kmkernel->xxx() constructs. The "kmkernel" is a define in kmkernel.h
declared as :
#define kmkernel KMKernel::self()
KMKernel is the central object in KMail. It's always created before
any other class, therefore you are _guaranteed_ that KMKernel::self()
(and therefore "kmkernel" construct) won't return 0 (null).
KMKernel implements the KMailIface (our DCOP interface) and gives
access to all the core KMail functionality.
IDENTITY
========
Files: identity*, kmidentity.{h,cpp}, configuredialog.cpp,
signatureconfigurator.{h,cpp}
Contact Marc Mutz <mutz@kde.org> on questions...
Identities consists of various fields represented by
QStrings. Currently, those fields are hardcoded, but feel free to
implement KPIM::Identity as a map from strings to QVariants or somesuch.
One part of identities are signatures. They can represent four modes
(Signature::Type) of operation (disabled, text from file or command
and inline text), which correspond to the combo box in the
identitydialog.
Identities are designed to be used through the KPIM::IdentityManager:
const KPIM::Identity & ident =
kmkernel->identityManager()->identityForUoidOrDefault(...)
Make sure you assign to a _const_ reference, since the identityForFoo
methods are overloaded with non-const methods that access a different
list of identities in the manager that is used while configuring. That
is known source of errors when you use identityForFoo() as a parameter
to a method taking const KPIM::Identity &.
WARNING: Don't hold the reference longer than the current functions
scope or next return to the event loop. That's b/c the config dialog
is not modal and the user may hit apply/ok anytime between calls to
function that want to use the identity reference. Store the UOID
instead if you need to keep track of the identity. You may also want
to connect to one of the KPIM::IdentityManager::changed() or ::deleted()
signals, if you want to do special processing in case the identity
changes.
Thus, in the ConfigureDialog, you will see non-const KPIM::Identity
references being used, while everywhere else (KMMessage,
IdentityCombo) const references are used.
The KPIM::IdentityCombo is what you see in the composer. It's a
self-updating combo box of KPIM::Identity's. Use this if you want the user
to choose an identity, e.g. in the folder dialog.
Ihe IdentityListView is what you see in the config dialog's identity
management page. It's not meant to be used elsewhere, but is DnD
enabled (well, at the time of this writing, only drag-enabled). This
is going to be used to dnd identities around between KNode and KMail,
e.g.
The SignatureConfigurator is the third tab in the identity
dialog. It's separate since it is used by the identity manager to
request a new file/command if the current value somehow fails.
FILTER
=======
Contact Marc Mutz <mutz@kde.org> on questions...
Filters consist of a search pattern and a list of actions plus a few
flags to indicate when they are to be applied (kmfilter.h).
They are managed in a QPtrList<KMFilter>, called KMFilterMgr. This
filter magnager is responsible for loading and storing filters
(read/writeConfig) and for executing them (process). The unique
instance of the filter manager is held by the kernel
(KMKernel::filterMgr()).
The search pattern is a QPtrList of search rules (kmsearchpattern.h) and a
boolean operator that defines their relation (and/or).
A search rule consists of a field-QString, a "function"-enum and a
"contents" or "value" QString. The first gives the header (or
pseudoheader) to match against, the second says how to match (equals,
consists, is less than,...) and the third holds the pattern to match
against.
Currently, there are two types of search rules, which are mixed
together into a single class: String-valued and int-valued. The latter
is a hack to enable <size> and <age in days> pseudo-header matching.
KMSearchRules should better be organized like KMFilterActions are.
A filter action (kmfilteraction.h) inherits from KMFilterAction or one
of it's convenience sub-classes. They have three sub-interfaces: (1)
argument handling, (2) processing and (3) parameter widget handling.
Interface (1) consists of args{From,As}String(), name() and
isEmpty() and is used to read and write the arguments (if any) from/to
the config.
Interface (2) is used by the filter manager to execute the action
(process() / ReturnCode).
Interface (3) is used by the filter dialog to allow editing of
actions and consists of name(), label() and the
*ParamWidget*(). Complex parameter widgets are collected in
kmfawidget.{h,cpp}.
A typical call for applying filters is
KMKernel::filterMgr()
foreach message {
KMFilterMgr::process():
}
CONFIGURE DIALOG
================
Files: configuredialog*.{h,cpp} ( identitylistview.{h,cpp} )
Contact Marc Mutz <mutz@kde.org> on questions...
The configuredialog is made up of pages that in turn may consist of a
number of tabs. The genral rule of thumb is that each page and tab is
responsible for reading and writing the config options presented on
it, although in the future, that may be reduced to interacting with
the corresponding config manager instead. But that won't change the
basic principle.
Thus, there is an abstract base class ConfigurePage (defined in
configuredialog_p.h), which derives from QWidget. It has four methods
of which you have to reimplement at least the first two:
- void setup()
Re-read the config (from the config file or the manager) and update
the widgets correspondingly. That is, you should only create the
widgets in the ctor, not set the options yet. The reason for that is
that the config dialog, once created, is simply hidden and shown
subsequently, so we need a reset-like method anyway.
- void apply()
Read the config from the widgets and write it into the config file
or the corresponding config manager.
- void installProfile()
This is called when the user selected a profile and hit apply. A
profile is just another KConfig object. Therefore, this method
should be the same as setup(), except that you should only alter
widgets for configs that really exist in the profile.
For tabbed config pages, there exists a convenience class called
TabbedConfigurationPage, which (as of this writing only offers the
addTab() convenience method. It is planned to also provide
reimplementations of setup, dismiss, apply and installProfile that just
call the same functions for each tab.
MDNs
====
Files: libkdenetwork/kmime_mdn.{h,cpp} and kmmessage.{h,cpp}, mostly
Contact Marc Mutz <mutz@kde.org> on questions...
MDNs (Message Disposition Notifications; RFC 2298) are a way to send
back information regarding received messages back to their
sender. Examples include "message read/deleted/forwarded/processed".
The code in kmime_mdn.{h,cpp} is responsible for creating the
message/disposition-notification body part (2nd child of
multipart/report that makes the MDN) and for providing the template
for human-readable text that goes into the text/plain part (1st child
of the multipart/report).
The code in KMMessage::createMDN() actually constructs a message
containing a MDN for this message, using the kmime_mdn helper
functions. It starts by checking the index for an already sent MDN,
since the RFC demands that MDNs be sent only once for every
message. If that test succeeds, it goes on to check various other
constraints as per RFC and if all goes well the message containing the
multipart/report is created.
If you need to use this functionality, see KMReaderWin::touchMsg() and
KMFilterAction::sendMDN() for examples. The touchMsg() code is invoked
on display of a message and sends a "displayed" MDN back (if so
configured), whereas the KMFilterAction method is a convenience helper
for the various filter actions that can provoke a MDN (move to trash,
redirect, forward, ...).
Folders
=======
Files: kmfolder*.{h,cpp} and *job.{h,cpp}
Contact Zack Rusin <zack@kde.org> with questions...
The inheritance hierarchy among KMail folder structure looks
as follows :
KMFolderNode
/ \
/ \
KMFolderDir \
KMFolder
|
KMFolderIndex
|
|
---< actual folder types: KMFolderImap, KMFolderMbox... >--
Besides the above mentioned classes one more relevant to our
discussion is KMFolderMgr which as the name suggest serves as the
folder manager.
At the base KMail's folder design starts with KMFolderNode which
inherits QObject. KMFolderNode is the base class encapsulating such
common folder properties as the name, boolean saying whether the
given folder is a directory storing the folders with mail or whether
the folder is a folder holding mail directly and finally a
KMFolderDir.
KMFolderDir is, as one might say, a directory abstration which manages
purely abstract KMFolder's. KMFolder's often don't have a disk based
image, they are entities existing only within KMail's design. To
manage the contents of one directory that may contain folders and/or
other directories KMFolderDir was created.
It inherits KMFolderNode and KMFolderNodeList. Yes, the inheritance is rather
bogus and has to be fixed especially if you'll notice that
KMFolderNode holds a pointer to the derived from it KMFolderDir.
KMFolderDir also inherits KMFolderNodeList which is a
QPtrList<KMFolderNode>. A special case of a KMFolderDir is known as
KMFolderRootDir and is supposed to represent the toplevel KMFolderDir
in the KMail's folder hierarchy.
KMFolderDir's serve as folder holders which are managed by
KMFolderMgr's. KMail contains three main KMFolderMgr's. They can be
accessed via KMKernel ( the "kmkernel" construct ). Those methods are :
1) KMFolderMgr *folderMgr() - which returns the folder manager for
the folders stored locally.
2) KMFolderMgr *imapFolderMgr() - which returns the folder manager
for all imap folders. They're handled a little differently because
for all imap messages only headers are cached locally while the
main contents of all messages is kept on the server.
3) KMFolderMgr *dimapFolderMgr() - which returns disconnected IMAP (dimap)
folder manager. In dimap, both the headers and a copy of the full message
is cached locally.
4) KMFolderMgr *searchFolderMgr() - which returns the folder manager
for the search folders (the folders created by using the "find
messages" tool. Other email clients call that type of folders
"virtual folders".
Finally one has to remember FolderJob classes. These classes allow
one to have asynchronous operations on KMFolder's. You simply create
a Job on a heap, connect to one of its signals and wait till the job
finishes. Folders serve as FolderJob's factories. For example to
retrieve the full message from a folder you'd have to do something like :
FolderJob *job = folderParent->createJob( aMsg, tGetMessage );
connect( job, SIGNAL(messageRetrieved(KMMessage*)),
SLOT(msgWasRetrieved(KMMessage*)) );
job->start();
Index (old)
===========
Files: kmfolderindex.{h,cpp} and kmmsg{base,info}.{h,cpp}
Contact Marc Mutz <mutz@kde.org> or
Till Adam <till@adam-lilienthal.de> or
Don Sanders <sanders@kde.org>
with questions...
index := header *entry
header := magic LF NUL header-length byte-order-marker sizeof-long
magic := "# KMail-Index V" 1*DIGITS
header-length := Q_UINT32
byte-order-marker := Q_UINT32( 0x12345678 )
sizeof-long := Q_UINT32( 4 / 8 )
entry := tag length value
tag := Q_UINT32 ; little endian (native?)
length := Q_UINT16 ; little endian (native?)
value := unicode-string-value / ulong-value
unicode-string-value := 0*256QChar ; network-byte-order
ulong-value := unsigned_long ; little endian
Currently defined tag values are:
Msg*Part num. val type obtained by:
No 0 u -
From 1 u fromStrip().stripWhitespace()
Subject 2 u subject().stripWhitespace()
To 3 u toStrip().stripWhiteSpace()
ReplyToIdMD5 4 u replyToIdMD5().stripWhiteSpace()
IdMD5 5 u msgIdMD5().stripWhiteSpace()
XMark 6 u xmark().stripWhiteSpace()
Offset 7 l folderOffset() (not only mbox!)
LegacyStatus 8 l mLegacyStatus
Size 9 l msgSize()
Date 10 l date()
File 11 u fileName() (not only maildir!)
CryptoState 12 l (signatureState() << 16) | encryptionState())
MDNSent 13 l mdnSentState()
ReplyToAuxIdMD5 14 u replyToAuxIdMD5()
StrippedSubject 15 u strippedSubjectMD5().stripWhiteSpace()
Status 16 l status()
u: unicode-string-value; l: ulong-value
Proposed new (KDE 3.2 / KMail 1.6) entry format:
index := header *entry
entry := sync 1*( tag type content ) crc-tag crc-value
sync := Q_UINT16 (32?) ; resync mark, some magic bit pattern
; (together with preceding crc-tag provides
; 24(40)bits to resync on in case of index
; corruption)
tag := Q_UINT8
type := Q_UINT8
content := variable-length-content / fixed-length-content
crc-tag := tag type ; tag=CRC16, type=CRC16
crc-value := Q_UINT16 ; the CRC16 sum is calculated over all of
; 1*( tag type content )
variable-length-content := length *512byte padding
padding := *3NUL ; make the string a multiple of 4 octets in length
fixed-length-content := 1*byte
length := Q_UINT16 (Q_UINT8?)
byte := Q_UINT8
The type field is pseudo-structured:
bit: 7 6 5 4 3 2 1 0
+-----+-----+-----+-----+-----+-----+-----+-----+
| uniq | chunk | len |
+-----+-----+-----+-----+-----+-----+-----+-----+
uniq: 3 bits = max. 8 different types with same chunk size:
for chunk = (0)00 (LSB(base)=0: octets):
00(0) Utf8String
01(0) BitField
10(0) reserved
11(0) Extend
for chunk = (1)00 (LSB(base)=1: 16-octet blocks):
00(1) MD5(Hash/List)
01(1) reserved
10(1) reserved
11(1) Extend
for chunk = 01 (shorts):
000 Utf16String
001-110 reserved
111 Extend
for chunk = 10 (int32):
000 Utf32String (4; not to be used yet)
001 Size32
010 Offset32
011 SerNum/UOID
100 DateTime
101 Color (QRgb: (a,r,g,b))
110 reserved
111 Extend
for chunk = 11 (int64):
000 reserved
001 Size64
010 Offset64
011-110 reserved
111 Extend
len: length in chunks
000 (variable width -> Q_UINT16 with the real width follows)
001..111: fixed-width data of length 2^len (2^1=2..2^6=128)
You find all defined values for the type field in indexentrybase.cpp
Currently defined tags are:
tag type content
DateSent DateTime Date:
DateReceived DateTime last Received:'s date-time
FromDisplayName String decoded display-name of first From: addr
ToDisplayName String dto. for To:
CcDisplayName String dto. for Cc:
FromAddrSpecs String possibly IMAA-encoded, comma-separated addr-spec's of From:
ToAddrSpecs String dto. for To:
CcAddrSpecs String dto. for Cc:
Subject String decoded Subject:
BaseSubjectMD5 String md5(utf8(stripOffPrefixes(subject())))
BodyPeek String body preview
MaildirFile String Filename of maildir file for this messagea
MBoxOffset Offset(64) Offset in mbox file (pointing to From_)
MBoxLength Size(64) Length of message in mbox file (incl. From_)
Size Size(64) rfc2822-size of message (in mbox: excl. From_)
Status BitField (see below)
MessageIdMD5 MD5Hash MD5Hash of _normalized_ Message-Id:
MDNLink SerialNumber SerNum of MDN received for this message
DNSLink SerialNumber SerNUm of DSN received for this message
ThreadHeads SerialNumberList MD5Hash's of all (so far discovered)
_top-level thread parents_
ThreadParents SerialNumberList MD5Hash's of all (so far discovered)
thread parents
"String" is either Utf8String or (Utf16String or Latin1String),
depending on content
Currently allocated bits for the Status BitField are:
Bit Value: on(/off) (\imapflag)
# "std stati":
0 New (\Recent)
1 Read (\Seen)
2 Answered (\Answered)
3 Deleted (\Deleted)
4 Flagged (\Flagged)
5 Draft (\Draft)
6 Forwarded
7 reserved
# message properties:
8 HasAttachments
9 MDNSent ($MDNSent)
10..11 00: unspecified, 01: Low, 10: Normal, 11: High Priority
12..15 0001: Queued, 0010: Sent, 0011: DeliveryDelayed,
0100: Delivered, 0101: DeliveryFailed,
0110: DisplayedToReceiver, 0111: DeletedByReceiver,
1001: ProcessedByReceiver, 1010: ForwardedByReceiver,
1011-1110: reserved
1111: AnswerReceived
# signature / encryption state:
16..19 0001: FullyEncrypted, 0010: PartiallyEncrypted, 1xxx: Problem
20..23 0001: FullySigned, 0010: PartiallySigned, 1xxx: Problem
# "workflow stati":
24..25 01: Important, 10: ToDo, 11: Later (from Moz/Evo)
26..27 01: Work, 10: Personal, 11: reserved (dto.)
28..29 01: ThreadWatched, 10: ThreadIgnored, 11: reserved
30..31 reserved
All bits and combinations marked as reserved MUST NOT be altered if
set and MUST be set to zero (0) when creating the bitfield.
Headers (Threading and Sorting)
===============================
Contact Till Adam <till@adam-lilienthal.de> or
Don Sanders <sanders@kde.org>
with questions...
Threading and sorting is implemented in kmheaders.[cpp|h] and headeritem.[cpp|h]
and involves a handfull of players, namely:
class KMHeaders:
this is the listview that contains the subject, date etc of each mail.
It's a singleton, which means there is only one, per mainwidget headers
list. It is actually a member of KMMainwidget and accessible there.
class HeaderItem:
these are the [Q|K]ListViewItem descendend items the KMHeaders listview
consists of. There's one for each message.
class SortCacheItem:
these are what the threading and sorting as well as the caching of those
operate on. Each is paired with a HeaderItem, such that each holds a
pointer to one, and vice versa, each header item holds a pointer to it's
associated sort cache item.
.sorted file:
The order of the sorted and threaded (if threading is turned on for this
folder) messages is cached on disk in a file named .$FOLDER.index.sorted
so if, for example the name of a folder is foo, the associated sorting
cache file would be called ".foo.index.sorted".
For each message, its serial number, that of its parent, the length of its
sorting key, and the key itself are written to this file. At the start of
the file several per folder counts and flags are cached additionally,
immediately after a short file headers. The entries from the start of the
file are:
- "## KMail Sort V%04d\n\t" (magic header and version string)
- byteOrder flag containing 0x12345678 for byte order testing
- column, the sort column used for the folder
- ascending, the sort direction
- threaded, is the view threaded or is it not?
- appended, have there been items appended to the file (obsolete?)
- discovered_count, number of new mail discovered since the last sort file
generation
- sorted_count, number of sorted messages in the header list
What is used for figuring out threading?
- messages can have an In-Reply-To header that contains the message id of
another message. This is called a perfect parent.
- additionally there is the References header which, if present, holds a
list of message ids that the current message is a follow up to. We
currently use the second to last entry in that list only. See further
down for the reasoning behind that.
- If the above two fail and the message is prefixed (Re: foo, Fwd: foo etc.)
an attempt is made to find a parent by looking for messages with the same
subject. How that is done is explained below as well.
For all these comparisons of header contents, the md5 hashes of the headers
are used for speed reasons. They are stored in the index entry for each
message. All data structures described below use md5 hash strings unless
stated otherwise.
Strategy:
When a folder is opened, updateMessageList is called, which in turn calls
readSortOrder where all the fun happens. If there is a .sorted file at the
expected location, it is openend and parsed. The header flags are read in
order to figure out the state in which this .sorted file was written. This
means the sort direction, whether the folder is threaded or not etc.
FIXME: is there currently sanity checking going on?
Now the file is parsed and for each message in the file a SortCacheItem is
created, which holds the offset in the .sorted file for this message as well
as it's sort key as read from the file. That sort cache item is entered into
the global mSortCache structure (member of the KMHeaders instance), which is
a QMemArray<SortCacheItem *> of the size mFolder->count(). Note that this
is the size reported by the folder, not as read from the .sorted file. The
mSortCache (along with some other structures) is updated when messages are
added or removed. More on that further down.
As soon as the serial number is read from the file, that number is looked up
in the message dict, to ensure it is still in the current folder. If not, it
has been moved away in the meantime (possibly by outside forces such as
other clients) and a deleted counter is incremented and all further
processing stopped for this message.
The messages parent serial number, as read from the sorted file is then
used to look up the parent and reset it to -1 should it not be in the
current folder anymore. -1 and -2 are special values that can show up
as parent serial numbers and are used to encode the following:
-1 means this message has no perfect parent, a parent for it needs to
be found from among the other messages, if there is a suitable one
-2 means this message is top level and will forever stay so, no need
to even try to find a parent. This is also used for the non-threaded
case. These are messages that have neither an In-Reply-To header nor
a References header and have a subject that is not prefixed.
In case there is a perfect parent, the current sort cache item is
appended to the parents list of unsorted children, or to that of
root, if there is not. A sort cache item is created in the mSortCache
for the parent, if it is not already there. Messages with a parent of
-1 are appended to the "unparented" list, which is later traversed and
its elements threaded. Messages with -2 as the parent are children of
root as well, as noted above, and will remain so.
Once the end of the file is reached, we should have a nicely filled
mSortCache, containing a sort cache item for each message that was in the
sorted file. Messages with perfect parents know about them, top level
messages know about that as well, all others are on a list and will be
threaded later.
Now, what happens when messages have been added to the store since the last
update of the .sorted file? Right, they don't have a sort cache item yet,
and would be overlooked. Consequently all message ids in the folder from 0
to mFolder->count() are looked at and a SortCacheItem is created for the
ones that do not have one yet. This is where all sort cache items are created
if there was no sorted file. The items created here are by definition un-
sorted as well as unparented. On creation their sort key is figured out as
well.
The next step is finding parents for those messages that are either new, or
had a parent of -1 in the .sorted file. To that end, a dict of all sort
cache items indexed by the md5 hash of their messsage id headers is created,
that will be used for looking up sort cache items by message id. The list of
yet unparented messages is then traversed and findParent() called for each
element wihch checks In-Reply-To and References headers and looks up the
sort cache item of those parents in the above mentioned dict. Should none be
found, the item is added to a second list the items of which will be subject
threaded.
How does findParent() work, well, it tries to find the message pointed to by
the In-Reply-To header and if that fails looks for the one pointed to by the
second to last entry of the References header list. Why the second to last?
Imagine the following situation in Bob's kmail:
- Bob get's mail from Alice
- Bob replies to Alice's mail, but his mail is stored in the Outbox, not the
Inbox.
- Alice replies again, so Bob now has two mails from Alice which are part of
the same thread. The mail in the middle is somewhere else. We want that to
thread as follows:
Bob <- In-Reply-To: Alice1
============
Alice1
|_Alice <- In-Reply-To: Bob (not here), References: Alice, Bob
- since the above is a common special case, it is worth trying. I think. ;)
If the parent found is perfect (In-Reply-To), the sort cache items is marked
as such. mIsImperfectlyThreaded is used for that purposer, we will soon see
why that flag is needed.
On this first pass, no subject threading is attempted yet. Once it is done,
the messages that are now top-level, the current thread heads, so to speak,
are collected into a second dict ( QDict< QPtrList< SortCacheItem > > )
that contains for each different subject an entry holding a list of (so far
top level) messages with that subject, that are potential parents for
threading by subjects. These lists are sorted by date, so the parent closest
by date can be chosen. Sorting of these lists happens via insertion sort
while they are built because not only are they expected to be short (apart
from hard corner cases such as cvs commit lists, for which subject threading
makes little sense anyhow and where it should be turned off), but also since
the messages should be roughly pre sorted by date in the store already.
Some cursory benchmarking supports that assumption.
If now a parent is needed for a message with a certain subject, the list of
top level messages with that subject is traversed and the first one that is
older than our message is chosen as it's parent. Parents more than six weeks
older than the message are not accepted. The reasoning being that if a new
message with the same subject turns up after such a long time, the chances
that it is still part of the same thread are slim. The value of six weeks
is chosen as a result of a poll conducted on #kde-devel, so it's probably
bogus. :) All of this happens in the aptly named: findParentBySubject().
Everthing that still has no parent after this ends up at toplevel, no further
attemp is made at finding one. If you are reading this because you want to
implement threading by body search or somesuch, please go away, I don't like
you. :)
Ok, so far we have only operated on sort cache items, nothing ui wise has
happened yet. Now that we have established the parent/child relations of all
messages, it's time to create HeaderItems for them for use in the header
list. But wait, you say, what about sorting? Wouldn't it make sense to do
that first? Exactly, you're a clever bugger, ey? Here, have a cookie. ;)
Both creation of header items and sorting of the as of yet unsorted sort
cache items happen at the same time.
As previously mentioned (or not) each sort cache item holds a list of its
sorted and one of its unsorted children. Starting with the root node the
unsorted list is first qsorted, and then merged with the list of already
sorted children. To achieve that, the heads of both lists are compared and
the one with the "better" key is added to the list view next by creating a
KMHeaderListItem for it. That header item receives both its sort key as well
as its id from the sort cache item. Should the current sort cache item have
children, it is added to the end of a queue of nodes to repeat these steps
on after the current level is sorted. This way, a breadth first merge sort
is performed on the sort cache items and header items are created at each
node.
What follows is another iteration over all message ids in the folder, to
make sure that there are HeaderItems for each now. Should that not be the
case, top level emergency items are created. This loop is also used to add
all header items that are imperfectly threaded to a list so they can be
reevalutated when a new message arrives. Here the reverse mapping from
header items to sort cache items are made as well. Those are also necessary
so the sort cache item based data structures can be updated when a message
is removed.
The rest of readSortOrder should make sense on itself, I guess, if not, drop
me an email.
What happens when a message arrives in the folder?
Among other things, the msgAdded slot is called, which creates the necessary
sort cache item and header item for the new message and makes sure the data
structures described above are updated accordingly. If threading is enabled,
the new message is threaded using the same findParent and findParentBySubject
methods used on folder open. If the message ends up in a watched or ignored
thread, those status bits are inherited from the parent. The message is also
added to the dict of header items, the index of messages by message id and,
if applicable and if the message is threaded at top level, to the list of
potential parents for subject threading.
After those house keeping tasks are performed, the list of as of yet imper-
fectly threaded messages is traversed and our newly arrived message is
considered as a new parent for each item on it. This is especially important
to ensure that parents arriving out of order after their children still end
up as parents. If necessary, the entries in the .sorted file of rethreaded
messages are updated. An entry for the new message itself is appended to the
.sorted file as well.
Note that as an optimization newly arriving mail as part of a mailcheck in
an imap folder is not added via msgAdded, but rather via complete reload of
the folder via readSortOrder(). That's because only the headers are gotten
for each mail on imap and that is so fast, that adding them individually to
the list view is very slow by comparison. Thus we need to make sure that
writeSortOrder() is called whenever something related to sorting changes,
otherwise we read stale info from the .sorted file. The reload is triggered
by the folderComplete() signal of imap folders.
What happens when a message is removed from the folder?
In this case the msgRemoved slot kicks in and updates the headers list. First
the sort cache item and header item representing our message are removed from
the data structures and the ids of all items after it in the store decre-
mented. Then a list of children of the message is assembled containing those
children that have to be reparented now that our message has gone away. If
one of those children has been marked as toBeDeleted, it is simply added to
root at top level, because there is no need to find a parent for it if it is
to go away as well. This is an optimization to avoid rethreading all
messages in a thread when deleting several messages in a thread, or even the
whole thread. The KMMoveCommand marks all messages that will be moved out of
the folder as such so that can be detected here and the useless reparenting
can be avoided. Note that that does not work when moving messages via filter
action.
That list of children is then traversed and a new parent found for each one
using, again, findParent and findParentBySubject. When a message becomes
imperfectly threaded in the process, it is added to the corresponding list.
The message itself is removed from the list of imperfectly threaded messages
and its header item is finally deleted. The HeaderItem destructor destroys
the SortCacheItem as well, which is hopefully no longer referenced anywhere
at this point.
DISPLAY (reader window - new)
====================================
Contact Marc Mutz <mutz@kde.org> with questions...
What happens when a message is to displayed in the reader window?
First, since KMMessagePart and KMMessage don't work with nested body
parts, a hierarchical tree of MIME body parts is constructed with
partNode's (being wrappers around KMMessagePart and
DwBodyPart). This is done in KMReaderWin::parseMsg(KMMessage*).
After some legacy handling, an ObjectTreeParser is instantiated and
it's parseObjectTree() method called on the root
partNode. ObjectTreeParser is the result of an ongoing refactoring
to enhance the design of the message display code. It's an
implementation of the Method Object pattern, used to break down a
huuuge method into many smaller ones without the need to pass around
a whole lot of paramters (those are made members
instead). parseObjectTree() recurses into the partNode tree and
generates an HTML representation of each node in the tree (although
some can be hidden by setting them to processed beforehand or - the
new way - by using AttachmentStrategy::Hidden). The HTML generation
is delegated to BodyPartFormatters, while the HTML is written to
HTMLWriters, which abstract away HTML sinks. One of those is
KHTMLPartHTMLWriter, which writes to the KHTMLPart in the
readerwindow. This is the current state of the code. The goal of the
ongoing refactoring is to make the HTML generation blissfully
ignorant of the readerwindow and to allow display plugins that can
operate against stable interfaces even though the internal KMail
classes change dramatically.
To this end, we designed a new set of interfaces that allows plugins
to be written that need or want to asynchronously update their HTML
representation. This set of interfaces consists of the following:
BodyPartFormatterPlugin
the plugin base class. Allows the BodyPartFormatterFactory to
query it for an arbitray number of BodyPartFormatters and
their associated metadata and url handlers.
BodyPartFormatter
the formatter interface. Contains a single method format()
that takes a BodyPart to process and a HTMLWriter to write the
generated HTML to and returns a result code with which it can
request more information or delegate the formatting back to
some default processor. BodyPartFormatters are meant to be
Flyweights, implying that the format() method is not allowed
to maintain state between calls, but see Mememto below.
BodyPart
body part interface. Contains methods to retrieve the content
of a message part and some of the more interesting header
information. This interface decouples the bodypart formatters
from the implementation used in KMail (or KNode, for that
matter). Also contains methods to set and retrieve a Memento,
which is used by BodyPartFormatters to maintain state between
calls of format().
BodyPartMemento
interface for opaque state storage and event handling.
Contains only a virtual destructor to enable the backend
processing code to destroy the object without the help of the
bodypart formatter.
During the design phase we identified a need for BodyPartFormatters to
request their being called on some form of events, e.g. a dcop
signal. Thus, the Memento interface also includes the IObserver and
ISubject interfaces. If a BodyPartFormatter needs to react to a signal
(Qt or DCOP), it implements the Memento interface using a QObject,
connects the signal to a slot on the Memento and (as an ISubject)
notifies it's IObservers when the slot is called. If a Memento is
created, the reader window registers itself as an observer of the
Memento and will magically invoke the corresponding BodyPartFormatter,
passing along the Memento to be retrieved from the BodyPart interface.
An example should make this clearer. Suppose, we want to update our
display after 10 seconds. Initially, we just write out an icon, and
after 10 seconds, we want to replace the icon by a "Hello world!"
line. The following code does exactly this:
class DelayedHelloWorldBodyPartFormatter
: public KMail::BodyPartFormatter {
public:
Result format( KMail::BodyPart * bodyPart,
KMail::HTMLWriter * htmlWriter ) {
if ( !bodyPart->memento() ) {
bodyPart->registerMemento( new DelayedHelloWorldBodyPartMemento() );
return AsIcon;
} else {
htmlWriter->write( "Hello, world!" );
return Ok;
}
}
};
class DelayedHelloWorldBodyPartMemento
: public QObject, public KMail::BodyPartMemento {
public:
DelayedHelloWorldBodyPartMemento()
: QObject( 0, "DelayedHelloWorldBodyPartMemento" ),
KMail::BodyPartMemento()
{
QTimer::singleShot( 10*1000, this, SLOT(slotTimeout()) );
}
private slots:
void slotTimeout() { notify(): }
private:
// need to reimplement this abstract method...
bool update() { return true; }
};
The contents of this file are now to be found and added to in Mainpage.dox.

869
Mainpage.dox
Unescape View File

@ -0,0 +1,869 @@
/** \mainpage KMail architectural overview
\section KMail design principles
This file is intended to guide the reader's way through the KMail
codebase. It should esp. be handy for people not hacking full-time on
KMail as well as people that want to trace bugs in parts of KMail
which they don't know well.
Contents:
- Kernel
- Identity
- Filters
- ConfigureDialog
- MDNs
- Folders
- Index
- Headers
- Display
TODO: reader, composer, messages, accounts, ...
\section kernel KERNEL
Files: kmkernel.h, kmkernel.cpp
Contact Zack Rusin <zack@kde.org> with questions...
The first thing you'll notice about KMail is the extensive use of
kmkernel->xxx() constructs. The "kmkernel" is a define in kmkernel.h
declared as :
#define kmkernel KMKernel::self()
KMKernel is the central object in KMail. It's always created before
any other class, therefore you are _guaranteed_ that KMKernel::self()
(and therefore "kmkernel" construct) won't return 0 (null).
KMKernel implements the KMailIface (our DCOP interface) and gives
access to all the core KMail functionality.
\section identity IDENTITY
FIXME this has moved to libkpimidentities, right?
Files: identity*, kmidentity.{h,cpp}, configuredialog.cpp,
signatureconfigurator.{h,cpp}
Contact Marc Mutz <mutz@kde.org> on questions...
Identities consists of various fields represented by
QStrings. Currently, those fields are hardcoded, but feel free to
implement KPIM::Identity as a map from strings to QVariants or somesuch.
One part of identities are signatures. They can represent four modes
(Signature::Type) of operation (disabled, text from file or command
and inline text), which correspond to the combo box in the
identitydialog.
Identities are designed to be used through the KPIM::IdentityManager:
const KPIM::Identity & ident =
kmkernel->identityManager()->identityForUoidOrDefault(...)
Make sure you assign to a _const_ reference, since the identityForFoo
methods are overloaded with non-const methods that access a different
list of identities in the manager that is used while configuring. That
is known source of errors when you use identityForFoo() as a parameter
to a method taking const KPIM::Identity &.
WARNING: Don't hold the reference longer than the current functions
scope or next return to the event loop. That's b/c the config dialog
is not modal and the user may hit apply/ok anytime between calls to
function that want to use the identity reference. Store the UOID
instead if you need to keep track of the identity. You may also want
to connect to one of the KPIM::IdentityManager::changed() or ::deleted()
signals, if you want to do special processing in case the identity
changes.
Thus, in the ConfigureDialog, you will see non-const KPIM::Identity
references being used, while everywhere else (KMMessage,
IdentityCombo) const references are used.
The KPIM::IdentityCombo is what you see in the composer. It's a
self-updating combo box of KPIM::Identity's. Use this if you want the user
to choose an identity, e.g. in the folder dialog.
Ihe IdentityListView is what you see in the config dialog's identity
management page. It's not meant to be used elsewhere, but is DnD
enabled (well, at the time of this writing, only drag-enabled). This
is going to be used to dnd identities around between KNode and KMail,
e.g.
The SignatureConfigurator is the third tab in the identity
dialog. It's separate since it is used by the identity manager to
request a new file/command if the current value somehow fails.
\section filter FILTER
Contact Marc Mutz <mutz@kde.org> on questions...
Filters consist of a search pattern and a list of actions plus a few
flags to indicate when they are to be applied (kmfilter.h).
They are managed in a QPtrList<KMFilter>, called KMFilterMgr. This
filter magnager is responsible for loading and storing filters
(read/writeConfig) and for executing them (process). The unique
instance of the filter manager is held by the kernel
(KMKernel::filterMgr()).
The search pattern is a QPtrList of search rules (kmsearchpattern.h) and a
boolean operator that defines their relation (and/or).
A search rule consists of a field-QString, a "function"-enum and a
"contents" or "value" QString. The first gives the header (or
pseudoheader) to match against, the second says how to match (equals,
consists, is less than,...) and the third holds the pattern to match
against.
Currently, there are two types of search rules, which are mixed
together into a single class: String-valued and int-valued. The latter
is a hack to enable \verbatim<size>\endverbatim and
\verbatim<age in days>\endverbatim pseudo-header matching.
KMSearchRules should better be organized like KMFilterActions are.
A filter action (kmfilteraction.h) inherits from KMFilterAction or one
of it's convenience sub-classes. They have three sub-interfaces: (1)
argument handling, (2) processing and (3) parameter widget handling.
Interface (1) consists of args{From,As}String(), name() and
isEmpty() and is used to read and write the arguments (if any) from/to
the config.
Interface (2) is used by the filter manager to execute the action
(process() / ReturnCode).
Interface (3) is used by the filter dialog to allow editing of
actions and consists of name(), label() and the
*ParamWidget*(). Complex parameter widgets are collected in
kmfawidget.{h,cpp}.
A typical call for applying filters is
KMKernel::filterMgr()
foreach message {
KMFilterMgr::process():
}
\section configuration CONFIGURE DIALOG
Files: configuredialog*.{h,cpp} ( identitylistview.{h,cpp} )
Contact Marc Mutz <mutz@kde.org> on questions...
The configuredialog is made up of pages that in turn may consist of a
number of tabs. The genral rule of thumb is that each page and tab is
responsible for reading and writing the config options presented on
it, although in the future, that may be reduced to interacting with
the corresponding config manager instead. But that won't change the
basic principle.
Thus, there is an abstract base class ConfigurePage (defined in
configuredialog_p.h), which derives from QWidget. It has four methods
of which you have to reimplement at least the first two:
- void setup()
Re-read the config (from the config file or the manager) and update
the widgets correspondingly. That is, you should only create the
widgets in the ctor, not set the options yet. The reason for that is
that the config dialog, once created, is simply hidden and shown
subsequently, so we need a reset-like method anyway.
- void apply()
Read the config from the widgets and write it into the config file
or the corresponding config manager.
- void installProfile()
This is called when the user selected a profile and hit apply. A
profile is just another KConfig object. Therefore, this method
should be the same as setup(), except that you should only alter
widgets for configs that really exist in the profile.
For tabbed config pages, there exists a convenience class called
TabbedConfigurationPage, which (as of this writing only offers the
addTab() convenience method. It is planned to also provide
reimplementations of setup, dismiss, apply and installProfile that just
call the same functions for each tab.
\section mdn MDNs
Files: libkdenetwork/kmime_mdn.{h,cpp} and kmmessage.{h,cpp}, mostly
Contact Marc Mutz <mutz@kde.org> on questions...
MDNs (Message Disposition Notifications; RFC 2298) are a way to send
back information regarding received messages back to their
sender. Examples include "message read/deleted/forwarded/processed".
The code in kmime_mdn.{h,cpp} is responsible for creating the
message/disposition-notification body part (2nd child of
multipart/report that makes the MDN) and for providing the template
for human-readable text that goes into the text/plain part (1st child
of the multipart/report).
The code in KMMessage::createMDN() actually constructs a message
containing a MDN for this message, using the kmime_mdn helper
functions. It starts by checking the index for an already sent MDN,
since the RFC demands that MDNs be sent only once for every
message. If that test succeeds, it goes on to check various other
constraints as per RFC and if all goes well the message containing the
multipart/report is created.
If you need to use this functionality, see KMReaderWin::touchMsg() and
KMFilterAction::sendMDN() for examples. The touchMsg() code is invoked
on display of a message and sends a "displayed" MDN back (if so
configured), whereas the KMFilterAction method is a convenience helper
for the various filter actions that can provoke a MDN (move to trash,
redirect, forward, ...).
\section folders Folders
Files: kmfolder*.{h,cpp} and *job.{h,cpp}
Contact Zack Rusin <zack@kde.org> with questions...
The inheritance hierarchy among KMail folder structure looks
as follows :
KMFolderNode
/ \
/ \
KMFolderDir \
KMFolder
|
KMFolderIndex
|
|
---< actual folder types: KMFolderImap, KMFolderMbox... >--
Besides the above mentioned classes one more relevant to our
discussion is KMFolderMgr which as the name suggest serves as the
folder manager.
At the base KMail's folder design starts with KMFolderNode which
inherits QObject. KMFolderNode is the base class encapsulating such
common folder properties as the name, boolean saying whether the
given folder is a directory storing the folders with mail or whether
the folder is a folder holding mail directly and finally a
KMFolderDir.
KMFolderDir is, as one might say, a directory abstration which manages
purely abstract KMFolder's. KMFolder's often don't have a disk based
image, they are entities existing only within KMail's design. To
manage the contents of one directory that may contain folders and/or
other directories KMFolderDir was created.
It inherits KMFolderNode and KMFolderNodeList. Yes, the inheritance is rather
bogus and has to be fixed especially if you'll notice that
KMFolderNode holds a pointer to the derived from it KMFolderDir.
KMFolderDir also inherits KMFolderNodeList which is a
QPtrList<KMFolderNode>. A special case of a KMFolderDir is known as
KMFolderRootDir and is supposed to represent the toplevel KMFolderDir
in the KMail's folder hierarchy.
KMFolderDir's serve as folder holders which are managed by
KMFolderMgr's. KMail contains three main KMFolderMgr's. They can be
accessed via KMKernel ( the "kmkernel" construct ). Those methods are :
1) KMFolderMgr *folderMgr() - which returns the folder manager for
the folders stored locally.
2) KMFolderMgr *imapFolderMgr() - which returns the folder manager
for all imap folders. They're handled a little differently because
for all imap messages only headers are cached locally while the
main contents of all messages is kept on the server.
3) KMFolderMgr *dimapFolderMgr() - which returns disconnected IMAP (dimap)
folder manager. In dimap, both the headers and a copy of the full message
is cached locally.
4) KMFolderMgr *searchFolderMgr() - which returns the folder manager
for the search folders (the folders created by using the "find
messages" tool. Other email clients call that type of folders
"virtual folders".
Finally one has to remember FolderJob classes. These classes allow
one to have asynchronous operations on KMFolder's. You simply create
a Job on a heap, connect to one of its signals and wait till the job
finishes. Folders serve as FolderJob's factories. For example to
retrieve the full message from a folder you'd have to do something like :
FolderJob *job = folderParent->createJob( aMsg, tGetMessage );
connect( job, SIGNAL(messageRetrieved(KMMessage*)),
SLOT(msgWasRetrieved(KMMessage*)) );
job->start();
\section index Index (old)
Files: kmfolderindex.{h,cpp} and kmmsg{base,info}.{h,cpp}
Contact Marc Mutz <mutz@kde.org> or
Till Adam <adam@kde.org> or
Don Sanders <sanders@kde.org>
with questions...
index := header *entry
header := magic LF NUL header-length byte-order-marker sizeof-long
magic := "# KMail-Index V" 1*DIGITS
header-length := Q_UINT32
byte-order-marker := Q_UINT32( 0x12345678 )
sizeof-long := Q_UINT32( 4 / 8 )
entry := tag length value
tag := Q_UINT32 ; little endian (native?)
length := Q_UINT16 ; little endian (native?)
value := unicode-string-value / ulong-value
unicode-string-value := 0*256QChar ; network-byte-order
ulong-value := unsigned_long ; little endian
Currently defined tag values are:
Msg*Part num. val type obtained by:
No 0 u -
From 1 u fromStrip().stripWhitespace()
Subject 2 u subject().stripWhitespace()
To 3 u toStrip().stripWhiteSpace()
ReplyToIdMD5 4 u replyToIdMD5().stripWhiteSpace()
IdMD5 5 u msgIdMD5().stripWhiteSpace()
XMark 6 u xmark().stripWhiteSpace()
Offset 7 l folderOffset() (not only mbox!)
LegacyStatus 8 l mLegacyStatus
Size 9 l msgSize()
Date 10 l date()
File 11 u fileName() (not only maildir!)
CryptoState 12 l (signatureState() << 16) | encryptionState())
MDNSent 13 l mdnSentState()
ReplyToAuxIdMD5 14 u replyToAuxIdMD5()
StrippedSubject 15 u strippedSubjectMD5().stripWhiteSpace()
Status 16 l status()
u: unicode-string-value; l: ulong-value
Proposed new (KDE 3.2 / KMail 1.6) entry format:
index := header *entry
entry := sync 1*( tag type content ) crc-tag crc-value
sync := Q_UINT16 (32?) ; resync mark, some magic bit pattern
; (together with preceding crc-tag provides
; 24(40)bits to resync on in case of index
; corruption)
tag := Q_UINT8
type := Q_UINT8
content := variable-length-content / fixed-length-content
crc-tag := tag type ; tag=CRC16, type=CRC16
crc-value := Q_UINT16 ; the CRC16 sum is calculated over all of
; 1*( tag type content )
variable-length-content := length *512byte padding
padding := *3NUL ; make the string a multiple of 4 octets in length
fixed-length-content := 1*byte
length := Q_UINT16 (Q_UINT8?)
byte := Q_UINT8
The type field is pseudo-structured:
bit: 7 6 5 4 3 2 1 0
+-----+-----+-----+-----+-----+-----+-----+-----+
| uniq | chunk | len |
+-----+-----+-----+-----+-----+-----+-----+-----+
uniq: 3 bits = max. 8 different types with same chunk size:
for chunk = (0)00 (LSB(base)=0: octets):
00(0) Utf8String
01(0) BitField
10(0) reserved
11(0) Extend
for chunk = (1)00 (LSB(base)=1: 16-octet blocks):
00(1) MD5(Hash/List)
01(1) reserved
10(1) reserved
11(1) Extend
for chunk = 01 (shorts):
000 Utf16String
001-110 reserved
111 Extend
for chunk = 10 (int32):
000 Utf32String (4; not to be used yet)
001 Size32
010 Offset32
011 SerNum/UOID
100 DateTime
101 Color (QRgb: (a,r,g,b))
110 reserved
111 Extend
for chunk = 11 (int64):
000 reserved
001 Size64
010 Offset64
011-110 reserved
111 Extend
len: length in chunks
000 (variable width -> Q_UINT16 with the real width follows)
001..111: fixed-width data of length 2^len (2^1=2..2^6=128)
You find all defined values for the type field in indexentrybase.cpp
Currently defined tags are:
tag type content
DateSent DateTime Date:
DateReceived DateTime last Received:'s date-time
FromDisplayName String decoded display-name of first From: addr
ToDisplayName String dto. for To:
CcDisplayName String dto. for Cc:
FromAddrSpecs String possibly IMAA-encoded, comma-separated addr-spec's of From:
ToAddrSpecs String dto. for To:
CcAddrSpecs String dto. for Cc:
Subject String decoded Subject:
BaseSubjectMD5 String md5(utf8(stripOffPrefixes(subject())))
BodyPeek String body preview
MaildirFile String Filename of maildir file for this messagea
MBoxOffset Offset(64) Offset in mbox file (pointing to From_)
MBoxLength Size(64) Length of message in mbox file (incl. From_)
Size Size(64) rfc2822-size of message (in mbox: excl. From_)
Status BitField (see below)
MessageIdMD5 MD5Hash MD5Hash of _normalized_ Message-Id:
MDNLink SerialNumber SerNum of MDN received for this message
DNSLink SerialNumber SerNUm of DSN received for this message
ThreadHeads SerialNumberList MD5Hash's of all (so far discovered)
_top-level thread parents_
ThreadParents SerialNumberList MD5Hash's of all (so far discovered)
thread parents
"String" is either Utf8String or (Utf16String or Latin1String),
depending on content
Currently allocated bits for the Status BitField are:
Bit Value: on(/off) (\\imapflag)
# "std stati":
0 New (\\Recent)
1 Read (\\Seen)
2 Answered (\\Answered)
3 Deleted (\\Deleted)
4 Flagged (\\Flagged)
5 Draft (\\Draft)
6 Forwarded
7 reserved
# message properties:
8 HasAttachments
9 MDNSent ($MDNSent)
10..11 00: unspecified, 01: Low, 10: Normal, 11: High Priority
12..15 0001: Queued, 0010: Sent, 0011: DeliveryDelayed,
0100: Delivered, 0101: DeliveryFailed,
0110: DisplayedToReceiver, 0111: DeletedByReceiver,
1001: ProcessedByReceiver, 1010: ForwardedByReceiver,
1011-1110: reserved
1111: AnswerReceived
# signature / encryption state:
16..19 0001: FullyEncrypted, 0010: PartiallyEncrypted, 1xxx: Problem
20..23 0001: FullySigned, 0010: PartiallySigned, 1xxx: Problem
# "workflow stati":
24..25 01: Important, 10: ToDo, 11: Later (from Moz/Evo)
26..27 01: Work, 10: Personal, 11: reserved (dto.)
28..29 01: ThreadWatched, 10: ThreadIgnored, 11: reserved
30..31 reserved
All bits and combinations marked as reserved MUST NOT be altered if
set and MUST be set to zero (0) when creating the bitfield.
\section headers Headers (Threading and Sorting)
Contact Till Adam <adam@kde.org> or
Don Sanders <sanders@kde.org>
with questions...
Threading and sorting is implemented in kmheaders.[cpp|h] and headeritem.[cpp|h]
and involves a handfull of players, namely:
class KMHeaders:
this is the listview that contains the subject, date etc of each mail.
It's a singleton, which means there is only one, per mainwidget headers
list. It is actually a member of KMMainwidget and accessible there.
class HeaderItem:
these are the [Q|K]ListViewItem descendend items the KMHeaders listview
consists of. There's one for each message.
class SortCacheItem:
these are what the threading and sorting as well as the caching of those
operate on. Each is paired with a HeaderItem, such that each holds a
pointer to one, and vice versa, each header item holds a pointer to it's
associated sort cache item.
.sorted file:
The order of the sorted and threaded (if threading is turned on for this
folder) messages is cached on disk in a file named .$FOLDER.index.sorted
so if, for example the name of a folder is foo, the associated sorting
cache file would be called ".foo.index.sorted".
For each message, its serial number, that of its parent, the length of its
sorting key, and the key itself are written to this file. At the start of
the file several per folder counts and flags are cached additionally,
immediately after a short file headers. The entries from the start of the
file are:
- "## KMail Sort V%04d\n\t" (magic header and version string)
- byteOrder flag containing 0x12345678 for byte order testing
- column, the sort column used for the folder
- ascending, the sort direction
- threaded, is the view threaded or is it not?
- appended, have there been items appended to the file (obsolete?)
- discovered_count, number of new mail discovered since the last sort file
generation
- sorted_count, number of sorted messages in the header list
What is used for figuring out threading?
- messages can have an In-Reply-To header that contains the message id of
another message. This is called a perfect parent.
- additionally there is the References header which, if present, holds a
list of message ids that the current message is a follow up to. We
currently use the second to last entry in that list only. See further
down for the reasoning behind that.
- If the above two fail and the message is prefixed (Re: foo, Fwd: foo etc.)
an attempt is made to find a parent by looking for messages with the same
subject. How that is done is explained below as well.
For all these comparisons of header contents, the md5 hashes of the headers
are used for speed reasons. They are stored in the index entry for each
message. All data structures described below use md5 hash strings unless
stated otherwise.
Strategy:
When a folder is opened, updateMessageList is called, which in turn calls
readSortOrder where all the fun happens. If there is a .sorted file at the
expected location, it is openend and parsed. The header flags are read in
order to figure out the state in which this .sorted file was written. This
means the sort direction, whether the folder is threaded or not etc.
FIXME: is there currently sanity checking going on?
Now the file is parsed and for each message in the file a SortCacheItem is
created, which holds the offset in the .sorted file for this message as well
as it's sort key as read from the file. That sort cache item is entered into
the global mSortCache structure (member of the KMHeaders instance), which is
a QMemArray<SortCacheItem *> of the size mFolder->count(). Note that this
is the size reported by the folder, not as read from the .sorted file. The
mSortCache (along with some other structures) is updated when messages are
added or removed. More on that further down.
As soon as the serial number is read from the file, that number is looked up
in the message dict, to ensure it is still in the current folder. If not, it
has been moved away in the meantime (possibly by outside forces such as
other clients) and a deleted counter is incremented and all further
processing stopped for this message.
The messages parent serial number, as read from the sorted file is then
used to look up the parent and reset it to -1 should it not be in the
current folder anymore. -1 and -2 are special values that can show up
as parent serial numbers and are used to encode the following:
-1 means this message has no perfect parent, a parent for it needs to
be found from among the other messages, if there is a suitable one
-2 means this message is top level and will forever stay so, no need
to even try to find a parent. This is also used for the non-threaded
case. These are messages that have neither an In-Reply-To header nor
a References header and have a subject that is not prefixed.
In case there is a perfect parent, the current sort cache item is
appended to the parents list of unsorted children, or to that of
root, if there is not. A sort cache item is created in the mSortCache
for the parent, if it is not already there. Messages with a parent of
-1 are appended to the "unparented" list, which is later traversed and
its elements threaded. Messages with -2 as the parent are children of
root as well, as noted above, and will remain so.
Once the end of the file is reached, we should have a nicely filled
mSortCache, containing a sort cache item for each message that was in the
sorted file. Messages with perfect parents know about them, top level
messages know about that as well, all others are on a list and will be
threaded later.
Now, what happens when messages have been added to the store since the last
update of the .sorted file? Right, they don't have a sort cache item yet,
and would be overlooked. Consequently all message ids in the folder from 0
to mFolder->count() are looked at and a SortCacheItem is created for the
ones that do not have one yet. This is where all sort cache items are created
if there was no sorted file. The items created here are by definition un-
sorted as well as unparented. On creation their sort key is figured out as
well.
The next step is finding parents for those messages that are either new, or
had a parent of -1 in the .sorted file. To that end, a dict of all sort
cache items indexed by the md5 hash of their messsage id headers is created,
that will be used for looking up sort cache items by message id. The list of
yet unparented messages is then traversed and findParent() called for each
element wihch checks In-Reply-To and References headers and looks up the
sort cache item of those parents in the above mentioned dict. Should none be
found, the item is added to a second list the items of which will be subject
threaded.
How does findParent() work, well, it tries to find the message pointed to by
the In-Reply-To header and if that fails looks for the one pointed to by the
second to last entry of the References header list. Why the second to last?
Imagine the following situation in Bob's kmail:
- Bob get's mail from Alice
- Bob replies to Alice's mail, but his mail is stored in the Outbox, not the
Inbox.
- Alice replies again, so Bob now has two mails from Alice which are part of
the same thread. The mail in the middle is somewhere else. We want that to
thread as follows:
Bob <- In-Reply-To: Alice1
============
Alice1
|_Alice <- In-Reply-To: Bob (not here), References: Alice, Bob
- since the above is a common special case, it is worth trying. I think. ;)
If the parent found is perfect (In-Reply-To), the sort cache items is marked
as such. mIsImperfectlyThreaded is used for that purposer, we will soon see
why that flag is needed.
On this first pass, no subject threading is attempted yet. Once it is done,
the messages that are now top-level, the current thread heads, so to speak,
are collected into a second dict ( QDict< QPtrList< SortCacheItem > > )
that contains for each different subject an entry holding a list of (so far
top level) messages with that subject, that are potential parents for
threading by subjects. These lists are sorted by date, so the parent closest
by date can be chosen. Sorting of these lists happens via insertion sort
while they are built because not only are they expected to be short (apart
from hard corner cases such as cvs commit lists, for which subject threading
makes little sense anyhow and where it should be turned off), but also since
the messages should be roughly pre sorted by date in the store already.
Some cursory benchmarking supports that assumption.
If now a parent is needed for a message with a certain subject, the list of
top level messages with that subject is traversed and the first one that is
older than our message is chosen as it's parent. Parents more than six weeks
older than the message are not accepted. The reasoning being that if a new
message with the same subject turns up after such a long time, the chances
that it is still part of the same thread are slim. The value of six weeks
is chosen as a result of a poll conducted on #kde-devel, so it's probably
bogus. :) All of this happens in the aptly named: findParentBySubject().
Everthing that still has no parent after this ends up at toplevel, no further
attemp is made at finding one. If you are reading this because you want to
implement threading by body search or somesuch, please go away, I don't like
you. :)
Ok, so far we have only operated on sort cache items, nothing ui wise has
happened yet. Now that we have established the parent/child relations of all
messages, it's time to create HeaderItems for them for use in the header
list. But wait, you say, what about sorting? Wouldn't it make sense to do
that first? Exactly, you're a clever bugger, ey? Here, have a cookie. ;)
Both creation of header items and sorting of the as of yet unsorted sort
cache items happen at the same time.
As previously mentioned (or not) each sort cache item holds a list of its
sorted and one of its unsorted children. Starting with the root node the
unsorted list is first qsorted, and then merged with the list of already
sorted children. To achieve that, the heads of both lists are compared and
the one with the "better" key is added to the list view next by creating a
KMHeaderListItem for it. That header item receives both its sort key as well
as its id from the sort cache item. Should the current sort cache item have
children, it is added to the end of a queue of nodes to repeat these steps
on after the current level is sorted. This way, a breadth first merge sort
is performed on the sort cache items and header items are created at each
node.
What follows is another iteration over all message ids in the folder, to
make sure that there are HeaderItems for each now. Should that not be the
case, top level emergency items are created. This loop is also used to add
all header items that are imperfectly threaded to a list so they can be
reevalutated when a new message arrives. Here the reverse mapping from
header items to sort cache items are made as well. Those are also necessary
so the sort cache item based data structures can be updated when a message
is removed.
The rest of readSortOrder should make sense on itself, I guess, if not, drop
me an email.
What happens when a message arrives in the folder?
Among other things, the msgAdded slot is called, which creates the necessary
sort cache item and header item for the new message and makes sure the data
structures described above are updated accordingly. If threading is enabled,
the new message is threaded using the same findParent and findParentBySubject
methods used on folder open. If the message ends up in a watched or ignored
thread, those status bits are inherited from the parent. The message is also
added to the dict of header items, the index of messages by message id and,
if applicable and if the message is threaded at top level, to the list of
potential parents for subject threading.
After those house keeping tasks are performed, the list of as of yet imper-
fectly threaded messages is traversed and our newly arrived message is
considered as a new parent for each item on it. This is especially important
to ensure that parents arriving out of order after their children still end
up as parents. If necessary, the entries in the .sorted file of rethreaded
messages are updated. An entry for the new message itself is appended to the
.sorted file as well.
Note that as an optimization newly arriving mail as part of a mailcheck in
an imap folder is not added via msgAdded, but rather via complete reload of
the folder via readSortOrder(). That's because only the headers are gotten
for each mail on imap and that is so fast, that adding them individually to
the list view is very slow by comparison. Thus we need to make sure that
writeSortOrder() is called whenever something related to sorting changes,
otherwise we read stale info from the .sorted file. The reload is triggered
by the folderComplete() signal of imap folders.
What happens when a message is removed from the folder?
In this case the msgRemoved slot kicks in and updates the headers list. First
the sort cache item and header item representing our message are removed from
the data structures and the ids of all items after it in the store decre-
mented. Then a list of children of the message is assembled containing those
children that have to be reparented now that our message has gone away. If
one of those children has been marked as toBeDeleted, it is simply added to
root at top level, because there is no need to find a parent for it if it is
to go away as well. This is an optimization to avoid rethreading all
messages in a thread when deleting several messages in a thread, or even the
whole thread. The KMMoveCommand marks all messages that will be moved out of
the folder as such so that can be detected here and the useless reparenting
can be avoided. Note that that does not work when moving messages via filter
action.
That list of children is then traversed and a new parent found for each one
using, again, findParent and findParentBySubject. When a message becomes
imperfectly threaded in the process, it is added to the corresponding list.
The message itself is removed from the list of imperfectly threaded messages
and its header item is finally deleted. The HeaderItem destructor destroys
the SortCacheItem as well, which is hopefully no longer referenced anywhere
at this point.
\section display DISPLAY (reader window - new)
Contact Marc Mutz <mutz@kde.org> with questions...
What happens when a message is to displayed in the reader window?
First, since KMMessagePart and KMMessage don't work with nested body
parts, a hierarchical tree of MIME body parts is constructed with
partNode's (being wrappers around KMMessagePart and
DwBodyPart). This is done in KMReaderWin::parseMsg(KMMessage*).
After some legacy handling, an ObjectTreeParser is instantiated and
it's parseObjectTree() method called on the root
partNode. ObjectTreeParser is the result of an ongoing refactoring
to enhance the design of the message display code. It's an
implementation of the Method Object pattern, used to break down a
huuuge method into many smaller ones without the need to pass around
a whole lot of paramters (those are made members
instead). parseObjectTree() recurses into the partNode tree and
generates an HTML representation of each node in the tree (although
some can be hidden by setting them to processed beforehand or - the
new way - by using AttachmentStrategy::Hidden). The HTML generation
is delegated to BodyPartFormatters, while the HTML is written to
HTMLWriters, which abstract away HTML sinks. One of those is
KHTMLPartHTMLWriter, which writes to the KHTMLPart in the
readerwindow. This is the current state of the code. The goal of the
ongoing refactoring is to make the HTML generation blissfully
ignorant of the readerwindow and to allow display plugins that can
operate against stable interfaces even though the internal KMail
classes change dramatically.
To this end, we designed a new set of interfaces that allows plugins
to be written that need or want to asynchronously update their HTML
representation. This set of interfaces consists of the following:
BodyPartFormatterPlugin
the plugin base class. Allows the BodyPartFormatterFactory to
query it for an arbitray number of BodyPartFormatters and
their associated metadata and url handlers.
BodyPartFormatter
the formatter interface. Contains a single method format()
that takes a BodyPart to process and a HTMLWriter to write the
generated HTML to and returns a result code with which it can
request more information or delegate the formatting back to
some default processor. BodyPartFormatters are meant to be
Flyweights, implying that the format() method is not allowed
to maintain state between calls, but see Mememto below.
BodyPart
body part interface. Contains methods to retrieve the content
of a message part and some of the more interesting header
information. This interface decouples the bodypart formatters
from the implementation used in KMail (or KNode, for that
matter). Also contains methods to set and retrieve a Memento,
which is used by BodyPartFormatters to maintain state between
calls of format().
BodyPartMemento
interface for opaque state storage and event handling.
Contains only a virtual destructor to enable the backend
processing code to destroy the object without the help of the
bodypart formatter.
During the design phase we identified a need for BodyPartFormatters to
request their being called on some form of events, e.g. a dcop
signal. Thus, the Memento interface also includes the IObserver and
ISubject interfaces. If a BodyPartFormatter needs to react to a signal
(Qt or DCOP), it implements the Memento interface using a QObject,
connects the signal to a slot on the Memento and (as an ISubject)
notifies it's IObservers when the slot is called. If a Memento is
created, the reader window registers itself as an observer of the
Memento and will magically invoke the corresponding BodyPartFormatter,
passing along the Memento to be retrieved from the BodyPart interface.
An example should make this clearer. Suppose, we want to update our
display after 10 seconds. Initially, we just write out an icon, and
after 10 seconds, we want to replace the icon by a "Hello world!"
line. The following code does exactly this:
class DelayedHelloWorldBodyPartFormatter
: public KMail::BodyPartFormatter {
public:
Result format( KMail::BodyPart * bodyPart,
KMail::HTMLWriter * htmlWriter ) {
if ( !bodyPart->memento() ) {
bodyPart->registerMemento( new DelayedHelloWorldBodyPartMemento() );
return AsIcon;
} else {
htmlWriter->write( "Hello, world!" );
return Ok;
}
}
};
class DelayedHelloWorldBodyPartMemento
: public QObject, public KMail::BodyPartMemento {
public:
DelayedHelloWorldBodyPartMemento()
: QObject( 0, "DelayedHelloWorldBodyPartMemento" ),
KMail::BodyPartMemento()
{
QTimer::singleShot( 10*1000, this, SLOT(slotTimeout()) );
}
private slots:
void slotTimeout() { notify(): }
private:
// need to reimplement this abstract method...
bool update() { return true; }
};
*/
Write Preview
Loading…
Cancel
Save