LD6 MAIL ARCHIVING – An Introduction

Although you can find all the relevant information in the Yellow books and the Red books, I found it scattered about in bits and pieces, and its even worse if you are also new to LD6. I created this to be a primer and a guide to setup the new Server based archiving that has been introduced in LD6. The project I am working on currently was the reason I got into this anyway, my team was entrusted to plan, test & setup an enterprise-wide archive solution using the inbuilt functionality of LD6. Mind you, with over 15,000 users and over 70 R5 servers all over the US, it is not going to be an easy task. We have finished all the testing and are now in the implementation stage. I will try and keep this document updated with any new findings and/ or corrections, from time to time.

This is by no means a comprehensive document, so I do recommend you read it only as a primer, and refer to the technical documentation for further details and how-tos. This document should basically help you understand the whole process and introduce you to LD6 archiving.

To implement LD6 Archiving, the following needs to be done: -

1. Upgrade to LD6 servers. Archiving needs the new LD6 NAB design, and all your mail files need to be upgraded to ODS 43 (and this by itself can be a large project).
2. Identify/ create archive servers with plenty of disk space for all those mail files.
3. Create policies for users.
4. Apply policies to users
5. Begin Archiving

Now in a little more detail.

Policies

Policies are central to LD6 Archiving, they are functionally similar to the Setup Profiles we have in R5 (and they still exist in R6), but a whole lot more powerful and customizable. Look it up if you want more information, but they are basically settings that can be applied to a person, a group, or even your whole organization.

Although there is a whole lot more to policies than just archiving, here we will only focus on that one aspect. But just for your information you can also use Policies for Registration, Setup, Desktop & Security settings.

Organizational & Explicit Policies

There are two types of policies, Organizational and Explicit. Organizational Policies (OPs) are assigned dynamically based on the hierarchal certificate they’re based on. Explicit Policies (EPs), as the name implies, are the Policies that are explicitly assigned to a set of users.

It took some time for me to get it through my thick head, so I reiterate, OPs are assigned dynamically, that means you don’t actually have to do anything to assign them to your users, except of course create the OP itself. They are calculated on the fly and applied to the users.

So if you have an OP called “/NYC/Acme” then it will dynamically apply to all users that have been registered with the NYC/ACME certificate.

Since OPs are what you would basically use to distribute settings throughout your organization, lets talk a little more about them. OPs test your organizational structure, so if you have been sloppy at creating and maintaining your certificates, you will be punished with a lot of prep work before you can go in this direction.

If you wish to maintain control over archiving in a granular fashion, you will need at least as many policies as your OU1s. Or you could have just one O level policy that applies to everybody. (A refresher: if you are identified as John Doe/Sales/NYC/Acme, then OU2=Sales, OU1=NYC & O=Acme).

EPs on the other hand have to be hard-coded in the person document for each user. In the LD6 NAB there is a field called “Policy” in the Person documents that can hold a single EP. Also EPs supersede OPs, and setup profiles supersede all policies (so its recommended that you not use them and use policies instead).

EPs are basically used wherever OPs cannot be easily applied. An EP would be useful, lets say, when you want all the Executives to have a different archiving criterion than the rest of the users. And since all your executives do not share the same certifier, you will probably not be able to use OPs for this task, whereas you can simply assign an EP to all the Executives and therefore treat them differently.

There are always exceptions to the rule, and in this case we have an Exception Policy for just those occasions. Exception Policies are used to override an OP or an EP that would otherwise apply to a user, so in this way the use get exempted from those particular settings.

Creating Policy documents

To create Policies you need a LD6 NAB on an LD6 Server as R5 Servers & Clients do not understand them. You also need the appropriate roles in the NAB to play with them (PolicyModifier, PolicyReader & PolicyCreator).

Each Policy Document requires a set of “Settings Document’ where all the configurations & rules are set. In the case of Archive Policies the Settings Document also requires a “Criteria Document” which has yet more settings. A Policy Document can have only one Settings Document, but a Settings Document can have multiple Criteria Documents.

This is what they look like: -

Policy document

Notice the other Setting Type besides Archiving here; those can also be applied using the same Policy document.

Archive Settings document

Probably the most important fields in the Settings document are the Source Server & the Destination Server.

Archive Criteria document

The Archive Criteria Settings’ document decides what & how to Archive.

Parents, Children, Inherit & Enforce

In the above images you must have also noticed the “Inherit” & “Enforce” fields. All the Policies can be configured in a hierarchical manner. So if you have user John Doe/Sales/NYC/Acme, his Policy could be derived from the following hierarchy: -

*/Acme                                              O level policy with global settings

*/NYC/Acme                               OU1 level policy for all New York users

*/Sales/NYC/Acme              OU2 level policy for all sales people in New York

So, depending on which OP is inheriting from which, John Doe’s Policy will have picked up different properties from each subsequent level. In a way, the lower the level (OU1, OU2) of the Policy the more influence it has. The */Sales/NYC/Acme policy could very well supercede the */NYC/Acme as well as the */Acme ones.

A Child Policy is one that inherits from another Policy. In the previous hierarchy */Sales/NYC/Acme is a child of */NYC/Acme is a child of */Acme. The lower Policies basically CAN inherit from the ones above it. In the case of EPs, since they are applied after all the other Policies, they too CAN inherit from other OPs. Although a user can only be assigned one EP at a time, one EP can inherit from another EP if they share the same hierarchy.

How the policy is derived in a hierarchy

OP Level I                                             

   OP Level II                                         

       OP Level III                                    

            ……

                EP Level I                             

                    EP Level II

                       EP Level III

                          ……..

In the above chart all the Policy settings in OP Level III will be passed on to the user except those marked “Inherit” in it, in which case those settings will be inherited from OP Level II. Another way the settings could come down from OP Level II is if it was marked as “Enforce”, in which case that setting will be enforced on OP Level III. OP Level II and OP Level I share the same relationship and OP Level II would be called a child of OP Level I, which would be the parent.

Now, the derived settings of the OPs trickle down to the EPs. EPs share the same relationship between themselves as the OPs, namely the parent-child relation. The difference is that EPs are always applied after the OPs have been. In effect you can use the “Enforce” option in a higher level Policy to apply the setting to the lower levels, or you could use the “Inherit” option in the lower level Policies to pull settings from their parents/ grandparents.

Of course, this is still a little too simplified, as you would probably need more than just a couple of Policies. You may want your Chicago users to archive documents older than 90 days, but your Boston users may need to keep their mail longer than that so you have a separate policy to archive documents only after 200 days. Or your Server Z is running out of disk space so you want all users on it to keep only 30 days worth of mail. As you can see planning policies can be a little more daunting, but for sure, you should crystallize on them before implementation, or you are asking for trouble later on.

If your head is already spinning there is good news, if you are with me so far then you’ve got most it already. I would say, planning/ creating Policies are 80% of the work involved in Archiving. Policies are the brains of this system and all the legwork is performed by the Compact task.

Tools

There are some great features in LD6 that help you manage policies;

You can create the archive criteria documents but not enable them and this lets you create all your policies and test them without actually archiving anything.

The Policy Synopsis Tool can generate detailed reports on which policy(s) apply to a user.

You also get an Assign Policy agent that will assign EPs to selected person documents.

There is also a view in the configuration pane of the Admin Client that gives additional information, but I did not get to use it much.

However there is no tool to monitor archiving progress and manage the actual archive mail files.

Archiving, what and how?

LD6 provides three options as to how many of the message is left behind in the user’s mail file.

1. None  -- That’s right, none. It copies the messages to the archive database and deletes them from the user’s mail file. To access the archived messages, the user will have to access the archive database.
   
   
2. Message Header only (Summary only as Lotus calls it) – This will copy the messages to the archive database but leave the header information behind in the user’s mail file. That means, fields like To, CC, BCC & Subject will still be there. This option also places a special icon in the mail views that indicate that the particular document has been archived. The message title also has an “Archived” appended to it (on a R5 client, however it will say “Truncated”).

   To go to the archived document, the user has an action button “Go to archive”, that open up the original document on the archive database, however R5 clients have no such button although you should be able to create one easily in the template mail template.

3. Message Header with 40K (Summary + 40K) – Same as the earlier option, except this will leave behind the first 40 KB of the message body too.

The choice is yours. You will need to balance the “mail file size” vs. “frustrate your users” factor. If there is too much mail left behind, performance of you servers will suffer. And on the other hand if too much mail is archived, then the users will be forced to repeatedly lookup on your archive servers causing network issues.

Regardless of the option you choose to archive, there is a cool setting that is enabled by default, which will not delete documents that have responses. This will archive all messages except any that are part of a thread, even if only one of them does not meet the archiving criteria. Responses and their parents are not archived till all the documents together do not meet the archiving criteria.

Note: Stationery & Calendar entries in the future are not archived.

Archiving, where to?

In R5, you had the option to archive your mail locally or on your home server. R6 (oops I meant LD6) provides a third option, to archive on a secondary server. And this is the option that makes LD6 archiving so interesting to us in the first place. You can have a bank of archive server with massive disk storage and dump all the old mail there. Yes, it will take up network bandwidth, but archiving will probably be done during off hours, so that should not be a problem. Would you trust all your organizations mail on that one Archive server (and Archiving is not cluster aware either)? Well you were thinking of backing up the archives further, on to tapes weren’t you? Even if the archive server dies for a 24 hour period, there not much to lose, as you can catch up the next night or in the weekend and best of all, users will probably not be crying too much about it either, (that’s unless your CEO suddenly decides he want one particular mail from 1995), but you get the general idea.

Ok, so we now have our hands on a big bad server, now what? Well if you have a whole lot of users on a whole lot of server in a whole lot of different locations, then you probably will be better off with more than just one archive server but that is not something we are going to get into here (I’m not going to do all the thinking for you J).

You can plan to keep all the archives in one directory, or make it more manageable by creating multiple directories, and of course this is controlled solely by the Archiving Policies you apply.

Once your Archive Server is ready, you should make yet another plan, a rollout plan. Lets say, you start by creating policies that archives all messages older than 30 days, you also have a large number of users, and you have enabled Archiving for all of them on a single day, that nigh compact is going to run, and continue running till it has archived your users’ millions of messages. So take it slow, first you need to get a feel of how many and how old the messages are in general in your users mail files. Then you can, lets say, apply a policy for 1500 days (a little more than 4 years) on Week 1, then bring it down by 200 – 300 days every week till you reach the magic figure. This will take time, but it will not overstretch your servers, and give you the window you need to fine tune your settings, and not to mention, clean up messes, which probably will happen no matter what.

How do you reach the magic figure? That should be a management decision, so later on when a user objects, you can blame it on them. But seriously, it depends on how your users use their mail. Sales people and Executives probably want all their mail at their fingertips, whereas your temp staff could probably care less about archiving.

Archiving, how does it happen?

The actual archiving takes place on the users home server (keep in mind that archiving could also be performed on another server where a replica of the mail file exists, i.e. cluster member). The Compact task is burdened with this job, and in LD6, it has new “-A” & “-a” switches especially for archiving. The uppercase ‘A’ does archiving but does not compact the mail file afterwards, but the ‘a’ stitch will.

You can run a console command “Load compact -a mail”, but a program document to run this on a scheduled basis would be more practical.

Multiple compact tasks don’t appear to be supported for archiving.

That’s all folks, Happy Archiving.

Keep in mind that;

• Policies need a LD6 NAB on an LD6 server to function. Your mail files and your archive server on the other hand, can be R5.

You can have only one EP per person document.

You cannot stop an R5 user from enabling local archiving. In the case of an LD6 client however, you can control local archiving if needed using Policies.

You should not allow both local and server archiving at the same time. You have only one set of messages; they can either be pushed to the local archive or to the server archive, not both.

When users move from one server to another, you will probably only need to move the archive file in accordance the new policy that applies to the user.

By default users can delete messages from their archives.

• The ACLs in the user’s mail file is copied over to the archived mail file, so delegations will still apply and they are always kept updated too.

The mail file owner field in the mail files need to be accurate for the policy to be applied correctly.

The archive mail files are copies and not replicas of the users’ mail files.

The Archive settings of a mail file are displayed incorrectly in an R5 client.

The Archive Profile Document maintains all the settings in a mail file.The compact task reads the NAB and derives the archive policy for each mail file it archives. The first time compact runs on a mail file, the “Owner” value of it is used to get the appropriate policy settings from the NAB.

There is no inbuilt tool available to check if all mail files are being archived, so you will probably have to create an application to monitor daily/weekly progress.

For further information refer to the following: -

1.       LD6 Admin help database or the Admin yellow books (also available in PDF versions at lotus.com)

2.       LD6 forums on Notes.net (http://www-10.lotus.com/ldd/nd6forum.nsf)

3.       Policy-based system administration LDD article

(Please send in your suggestions/ corrections to [email protected]. You may freely make copies/ prints of this document provided you acknowledge the source. The original and updated version of this document is available at www.geocities.com/neerajjain/archive.html)
(--R4 06/10/03--)