Strategies and Tools for Managing GroupWise ® Archives Greg M. Smith, Director of Technical...

Strategies and Tools for Managing GroupWise® Archives

Greg M. Smith, Director of Technical Services, Messaging Architectsgreg.smith@gwtools.com

Dean WalterPresident, Email Essentials Inc.dwalter@emailessentials.ca

© February 9, 2004 Novell Inc.2

Messaging Architects – Quick Intro…

• For over 8 years, a leading developer of innovative applications products that help Enhance, Secure, and Accelerate GroupWise®.

• Novell’s exclusive Premier Level DeveloperNet partner for GroupWise and NetMail®

• World class development and engineering resources dedicated to Messaging & Collaboration.

• A Trusted Advisor that can assist with planning, deploying, managing and supporting mission-critical Email systems & applications.

© February 9, 2004 Novell Inc.3

Agenda

• Overview

• Data Retention Necessities

• GroupWise Archive Architecture

• Deploying GroupWise Archives

• Managing GroupWise Archives

© February 9, 2004 Novell Inc.4

Some Sobering Facts…

StorageThe average user will attempt to retain/store 500 MB of messages this year

VolumeIDC projects 33 billion MPD in 2005, up from the current 23 billion

CostThe White House spent $10M to recover 246K messages from 4,900 backup tapes

© February 9, 2004 Novell Inc.5

Some Sobering Facts...

KnowledgeIDC reports that 60% of business critical information is stored in email systems.

Access

80% of archived data is not accessible in a timely or cost effective manner, impacting the organization's performance & productivity.

BackupsRestoration from tape is not always a certainty, information is often lost or requires substantial effort to recover.

© February 9, 2004 Novell Inc.6

Driving Factors

GroupWise Message Bloat• Increased Backup and Disaster Recovery Times• Increased Maintenance Costs• Increased potential for database problems• Maintenance routines cannot be run regularly

Information Overload

© February 9, 2004 Novell Inc.7

Driving Factors

Secure Message Access• Archived messages are encrypted• Messages include GroupWise Digital

Signature• Non-repudiation of message is

maintained

Data Integrity

© February 9, 2004 Novell Inc.8

Data Retention Needs

Driving Factors

Legal Requirements• Litigation (14% of companies have been

ordered to produce employee emails)• Personal harassment lawsuits increasing in

number and claims for damages

© February 9, 2004 Novell Inc.9

Driving Factors

Regulatory Requirements• HIPPA / Sarbanes-Oxley / Patriot

Act /SEC 17-4a• Federal/State/Municipal Guidelines &

Statutes• Access to Information

Big Brother

SEC 17-4a Regulations Governing Financial Institutions... Originals of all communications received and copies of all communications sent by such member, broker or dealer (including inter-office memoranda and communications)...

© February 9, 2004 Novell Inc.10

Why GroupWise Archives?

• Messages are retained in GroupWise format• Message integrity and non-repudiation retained• Message audit trail and delivery status’ maintained• Access integrated into GroupWise client• Messages can be seamlessly moved between

archives and live system• Messages are fully indexed and searchable• Same Look and Feel for users for ease of use

Accessibility

© February 9, 2004 Novell Inc.11

ArchitectureAnatomy of an Archive

Archive Structure Similar to Post Office:

MSG.DB

OFxxxARC

USER.DB

BLOBATTACHMENTS

© February 9, 2004 Novell Inc.12

ArchitectureAnatomy of an archive

GroupWise Client Managed• Windows Client

• Manages archive based on FID

• Needs only root directory

• Can handle multiple archives in separate archive root directories

• Integrated GWCheck Management

OFxxxARC

OFyyyARC

OFzzzARC

Archive Root

© February 9, 2004 Novell Inc.13

Deployment ConsiderationsChoosing Where to Archive

Network location• Higher Data Integrity• Centralized management• Secure with file rights• GroupWise Client Options

Workstations• Excess local capacity• Access Limitations• Lower data integrity

© February 9, 2004 Novell Inc.14

Deployment ConsiderationsStorage Requirements

Increased Storage• Messages are compressed & Encrypted• Shared Attachments are stored for each user• Expect to double or triple current storage

© February 9, 2004 Novell Inc.15

Deployment ConsiderationsPolicies

Archiving Policies• Message retention policy

• Manual or forced archiving?

• Corresponding deletion policy?

• Archive business or transient email?

© February 9, 2004 Novell Inc.16

Deployment ConsiderationsArchive Accessibility

Accessibility • Portable (Live or Remote/Caching

Mode)

• File system access to archive path

• No web based access at this time

• No R/O format such as CDROM

Archive Management/Maintenance

© February 9, 2004 Novell Inc.18

Archive ManagementDatabase Maintenance

….GroupWise Archives are still relational databases and will require the same type of maintenance, upkeep and troubleshooting as the GroupWise Post Office……

© February 9, 2004 Novell Inc.19

Archive ManagementDatabase Maintenance

Large archives require periodic maintenance

• Admin GWCHK32 WIN32 Maintenance Utility• Perform Structural Checks • Perform Content Checks

• User Based Check / Rebuild Utility• CTRL-SHIFT Select File / Open Archive• Runs Analyze Check/Fix followed by Contents

Check/Fix• Runs a Structural Rebuild if uncorrectable errors are

encountered.

© February 9, 2004 Novell Inc.20

Archive ManagementDatabase Maintenance

Large archives require periodic maintenance

• Administrative Utilities GWCHK32.EXE • WIN32 Maintenance Utility• Perform Structural and Content Checks • Reduce and Expire Archive Records• Run Statistical Checks• Rebuild Databases

© February 9, 2004 Novell Inc.21

Archive ManagementDatabase Maintenance

Large archives require periodic maintenance

• User based maintenance utility• CTRL-SHIFT Select File | Open Archive• Runs Analyze Check/Fix followed by Contents

Check/Fix• Runs a Structural Rebuild if uncorrectable errors are

encountered.• Completely automated with no user input required• Ideal for GroupWise remote and local archives

© February 9, 2004 Novell Inc.22

Archive ManagementDatabase Maintenance

Third Party Tools

• Email Essentials Inc.• CheckIT

• Runs GWCHCK32 operations against multiple archive databases

• Can perform up to four different operations per job• Configurations can be saved and run as batch jobs• Configurations can be exported and run

automatically at GroupWise Client startup

© February 9, 2004 Novell Inc.23

Archives MaintenanceArchive Access

• Archive is encrypted to individual user• What if user account is deleted?• What if users FID changes?• What if I need to access the archive data?

• FID Editor & FIDit (email essentials)• Allows administrator to change encryption of

archive to another user ID• Command Line based or GUI based (FIDit)

© February 9, 2004 Novell Inc.24

Archives MaintenanceArchive Maintenance

Archived Information Recovery

• MergeIT (email essentials)• Combines multiple user archives into a single

archive• Eliminates the need for de-archiving and re-

archiving

• GWArchive (Messaging Architects)• Administratively restore multiple Archives back into

their original accounts• Server based export of Archives & Mailboxes into an

externally readable and viewable format (XML) for one or multiple accounts. Includes full search & indexing capabilities. Can easily be burned to CD/DVD for portability and offline storage (retention)

© February 9, 2004 Novell Inc.25

Retention Policy Management

• GWCheck / CheckIT• Provides record deletion based on message

age

• GWArchive (Messaging Architects)• Policy engine that provides enhanced

database management for native GW Archives. Structure can be based on user-defined criteria (year/project/other). Offers improved system performance & stability + improved user interface.

• Provides server based publish/export to either external DB (MySQL, Oracle, MS-SQL, DB2) or to XML

Archives MaintenanceArchive Maintenance

© February 9, 2004 Novell Inc.26

Archives MaintenanceArchive Maintenance

Archive Data Management• HSM Products

• Caminosoft's Managed Server• A component of the GWArchive

solution

Allows either live OFFILES content or archived information from any mapped drive to be migrated to near-line storage, tape & NAS. Avoids the need to constantly upgrade server storage and can cut backup or recovery time by 50% or more.

© February 9, 2004 Novell Inc.27

• We are a private college in Dubuque, Iowa• GroupWise 6.5.1 system with 1500 mailboxes• Receive approximately 20,000 messages per day• Faculty, staff and students wanted the spam stopped• Junk folders are enabled but it wasn’t enough• 1st spam solution didn’t stop even half and RBLs aren’t

enough• We wanted a system we didn’t have to constantly

update/maintain• Needed a system with low false-positives

Clarke College Spam Solution

Hands down best solution: GWGuardian AV 3.0

© February 9, 2004 Novell Inc.28

CarlIT Department

JeffStudent

JaneAdmin. Assistant

EllenFaculty Member

NoneAveraged 5 to 10 spam e-mails per day

1 or 2 per monthAveraged 15 to 20 spam e-mails per day

At most 2 per day (Usually none)

Averaged 25 spam e-mails per day

Less than 5 per dayAveraged 150 spam e-mails per day

Clarke College Spam Examples

Before Implementation After ImplementationUser

© February 9, 2004 Novell Inc.29

In Conclusion

• Data Retention is clearly a major concern at all levels of industry and government

• GroupWise archives provide a viable method of retaining corporate messages and complying with organizational policies, but with clear limitations

• Application-independence and format-neutrality (i.e. XML + plain text) are critical attributes for any data destined to reside in long term storage (5+ years).

• Third party tools allow organizations to properly deploy and manage both retention/deletion policies and the resulting data sets that are generated as a result of these policies.

© February 9, 2004 Novell Inc.31

General DisclaimerThis document is not to be construed as a promise by any participating company to develop, deliver, or market a product. Novell, Inc., makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. Further, Novell, Inc., reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All Novell marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.

No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of Novell, Inc. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability.