Upload
panagenda
View
1.181
Download
0
Tags:
Embed Size (px)
DESCRIPTION
A IBM Connect 2013 session by David O'Neal (Consultant, Infrastructure Engineering, Nationwide), Wouter Aukema (CTO, Trust Factory) and Florian Vogler (CEO, panagenda)
Citation preview
© 2013 IBM Corporation
SPN101 Leveraging the Power ofIBM Lotus Notes and Dominoat NationwideDavid O’Neal | Consultant, Infrastructure Engineering, NationwideWouter Aukema | CTO, Trust FactoryFlorian Vogler | CEO, panagenda
What we‘ll cover today
Introduction
IBM Notes and Domino @ Nationwide
What we did– Collect Data (what sources & some stats)
What we found– Confirmations & Opportunities– Configuration, Usage, Performance, Security
What it means– Short Term Quick Wins– Long Term Strategic Takeaways
Conclusions / What we learned
Q&A
2
Introduction – who‘s who?
About Nationwide– Nationwide Mutual Insurance Company, based in Columbus, Ohio, is one of the largest
and strongest diversified insurance and financial services organizations in the U.S. and is rated A+ by both A.M. Best and Standard & Poor’s. The company provides customers a full range of insurance and financial services, including auto insurance, motorcycle, boat, homeowners, pet, life insurance, farm, commercial insurance, annuities, mortgages, mutual funds, pensions, long-term savings plans and specialty health services.
About Trust Factory– Trust Factory‘s DNA provides true insight into server performance and scaling
opportunities. DNA is also used by IBM worldwide as Domino DoubleCheck.
About panagenda– With more than 5.5 million licenses of its products, panagenda helps customers in over
70 countries analyze and optimize their IBM environments.
3
IBM Notes and Domino @ Nationwide
The Nationwide Notes/Domino Environment
– Production use began in 1997 with version 3.3– Migrated/Consolidated to Notes from cc:Mail and a variety of different mainframe email
systems – Current environment
• 6 Domino Domains• 200+ Domino servers on Microsoft Windows® (8.x mixture – mostly 8.5.3 for mail)
(Mail, Management, Application, Blackberry and Good servers)
• Active / Passive clustering accross two data centers• 56,000+ Notes clients (mostly 8.5.2)• 15,000,000+ messages routed weekly• ~20 Sametime 8.5.2 IFR1 servers using Domino and WebSphere• ~1200-1400 Domino applications with 700 being active
4
IBM Notes and Domino @ Nationwide
What is Nationwide trying to accomplish by performing this in-depth analysis?
– Server:• Discover inconsistant configurations, and find gaps where Domino does not readily
report items that could potentially turn into problems.
– Client:• Discover and inventory client side settings, configurations and local databases to get
a better understanding of client health and functionality.
– Environmental:• Combine server and client findings to get a holostic view of our Notes/Domino
environment.
5
What we did
Collect data from Domino servers– statrep.nsf– log.nsf– catalog.nsf– directories (names.nsf,
DA)
Inventory Notes clients– notes.ini– desktop, bookmarks,
names– local databases– various OS and Notes
properties
6
Talk
What we did
Collect– log, statrep and catalog from 151 servers– 33,000 users used 35,000+ clients– 690,000+ documents with 315,000+ attachments collected = 3,5 GB of raw data– 1.6 million desktop icons, 1.5 million local databases, 5.4 million notes.ini entries,
8.5 million client and OS details
Analyze– DNA: Compared this engagement against 2+ million other users– 100+ views created consuming 30+ GB of disk space
Interpret and Correlate
… and now for the meat …
7
12 April 2023
8
Domino Environment Overview
1 Domino Directory
39,725 Users Registered
153 Servers Registered
39,369 Groups Registered
34,057 Users Active 47,178 Databases Touched
151 Servers Analyzed 133,540 Databases Deployed
4 Domino Releases
4 Operating Systems
82,131 Db Storage (GB)
1,477,390 Views Defined
381 View Storage (GB)
494,006 Views Indexed
1,361,855 ACL Entries
1,370,468 Group Members
12 April 2023
9
DNA BenchmarkActive versus Registered Users
0 %
20 %
40 %
60 %
80 %
100 %
Nationwide Lowest Customer DNA Average Highest Customer
Unused Licenses, Web Users, Regular Absense
34,057 active users
12 April 2023
10
DNA BechmarkTime Online
-
5
10
15
20
25
Ses
sion D
ura
tion
(min
s per
ses
sion)
-
10
20
30
40
50
60
Online
Tim
e
(hours
per
use
r)
Session Duration 3 1 4 22
Online Time 24 2 23 77
Nationwide Lowest Customer DNA Average Highest Customer
On average with DNA
12 April 2023
11
User Demand Profiling(Nationwide, 34,057 active accounts)
0%
5%
10%
15%
20%
25%
2 4 6 8 10 12 14 16 18 20 22 24
Distinct Hours Online per Day
System AccountsOffice WorkersRemote Workers
12 April 2023
12
End User Demand Characteristics
Nationwide
0%
25%
50%
75%
100%
check new mail 6% 0% 0% 1% 0% 0%
system dbs 6% 0% 0% 1% 0% 0%
mail files 80% 85% 79% 85% 86% 99%
directories 3% 1% 5% 4% 0% 0%
applications 5% 14% 16% 9% 14% 1%
Notes Sessions Document Reads Document Writes Db Transactions Network Traffic Session Duration
Only mail servers in Scope for DNA
End User Demand Characteristics
Other IBM Customer
0%
25%
50%
75%
100%
check new mail 19% 0% 0% 3% 0% 1%
system dbs 13% 4% 0% 5% 1% 1%
mail files 33% 24% 76% 54% 55% 72%
directories 16% 41% 3% 11% 11% 4%
applications 19% 31% 22% 27% 33% 23%
Notes SessionsDocument ReadsDocument WritesDb TransactionsNetwork TrafficSession Duration
12 April 2023
13
Extreme high docreads on Directory databases
12 April 2023
14
0
1
10
100
1.000
10.000
100.000
1.000.000
10.000.000
100.000.000
1 10 100 1.000 10.000 100.000 1.000.000 10.000.000 100.000.000
Kilo
Byte
s Se
nt t
o Se
rver
.
KiloBytes Read from Server
User Demand on 16739, DatabasesNationwide
369, Application Dbs 560, Domino Directory Dbs 15209, Mailfiles 55, Mailin databases 143, Server Mail Boxes 403, System databases
Showing only databases touched by >1 users.(47,175 databases touched by all users)
Majority of apps are MC
12 April 2023
15
End User Demand at NationwideClassified by Demand Level
0% 25% 50% 75% 100%
User Sessions
Network Traffic(server to client)
Network Traffic(client to server)
DatabaseTransactions
DocumentReads
DocumentWrites
Extreme (1) Intensive (16) Moderate (804) Light (33,236)
1 user does 15% of total network demand
12 April 2023
16
Domino Servers at NationwideClassified by Maximum Session Concurrency
0
5
10
15
20
25
30
35
40
45
50
55
60
65
70
75
80
85
90
95
87 23 23 17 1
Very Low< 50
Low50 - 249
Average250 - 749
Normal750 - 1499
High>= 1500Level
Servers
Redistributing the load can reduce nr. of servers with up to 87
12 April 2023
17
End User DemandSession Concurrency
0
5,000
10,000
15,000
20,000
25,000
30,00020
12-1
2-04
00 04
08
12
16
20
2012
-12-
05 0
0 04
08
12
16
20
2012
-12-
06 0
0 04
08
12
16
20
2012
-12-
07 0
0 04
08
12
16
20
2012
-12-
08 0
0 04
08
12
16
20
2012
-12-
09 0
0 04
08
12
16
20
2012
-12-
10 0
0 04
08
12
16
20
Concu
rren
t U
ser Ses
sions
Max Observed Maximum: 27,179
12 April 2023
18
How Much is Notes Network Compression Used?
Enabled75%
Disabled25%
Includes Traffic from Users and Servers
Network Compression
# Users making use of
Notes Network Compression
0%
25%
50%
75%
100%
Persons Servers
% A
ctive
Use
rs
Enabled
Disabled
Very few customers have this properly
implemented
12 April 2023
19
Deployment Integrity
11 Group Cycles Detected
Integrity check # Databases
Duplicate Replica On Same Server 380
Duplicate Template On Same Server 341
Replicas Acting As Different Template 610
Same Replica But Different Inheritance -
Grand Total 1,331
Document Type Item Type Nr of Documents
group docs listname 3
mail-in docs fullname 22
person docs fullname 2
Grand Total 27
Entries appearing in multiple documents
PubNames, DirCat & DA at
risk (!)
12 April 2023
20
Basic Security Checks
Access Level Databases Templates
Author 84 102
Editor 11 -
Manager 3 302
Reader 2,507 222
Grand Total 2,605 626
Databases with Anonymous Access
1st Customer with NO issues :)
Variations found Accounts
'password' 0
'secret' 0
firstname 0
lastname 0
shortname 0
companyname 0
Grand Total 0
Internet Password Strength
And now for the client-side analysis …
21
Diving right into client-side analysis
The following slides dive intovarious client-side details
In many cases, the Nationwideenvironment is surprisingly clean
– Your environment willmost probably look very different
22
Notes 8.0.2 & 8.5.2
Although there are 1,817 clients with 8.0.2,only 26 have Create_R8_Databases enabled =they do not leverage the benefits of ODS 48
23
Operating system details
Various different client-side operating systems in use:
24
Local replicas of public addressbook
Local replicas of the public addressbook beyond cutoff
Risk of replicating deleted documentsback into server-side replica
Enable PIRC:
25
Local addressbooks:Version mismatch
Checked rows show configurationswhere names design matchesclient version
– (might still have wrong ODS, though)
In general, design mismatch ofsystem databases
– slows down client startup and beyond– causes unexpected behaviour or
non-functioning of Policies
can be fixed by– making sure clients have correct templates– removing TemplateSetup= from notes.ini
26
Local bookmark.nsfs:Version mismatch
Checked rows show configurationswhere bookmark design matchesclient version
– (might still have wrong ODS, though)
27
Local cache.ndk:Version mismatch
Checked rows show configurationswhere cache design does NOT matchclient version
– (might still have wrong ODS, though)
Cache.ndk must be deleted andre-created together withCREATE_R85_DATABASES=1in notes.ini - for it to have properdesign and ODS(make sure client has correct cache.ntf)
28
Local desktop.ndk:ODS issues
Checked rows show configurationswhere desktop ODS is NOT ideal
29
Local log.nsfs:Version mismatch
Checked rows show configurationswhere log design does NOT matchclient version
– (might still have wrong ODS, though)
30
Notes.ini:Log=
A couple of users have multiple log= lines in notes.ini
Since only the first entry is actually read in such a case,logging does not work as expected for those users
31
More on ODS levels
Various databases and templates do not havean ideal ODS …:
AddingCREATE_R85_DATABASES=1andNSF_UpdateODS=1to notes.ini can help!
32
More notes.ini entries …
Less than 1% of all users have port compressiondisabled, but 25% of all traffic is uncompressed must be enabled on BOTH servers and clients identify servers that are used by users but haveport compression disabled
33
EXTMGR_ADDINS= …
Various users haveEXTMGR_ADDINSentries in notes.ini which areseperated by a blank
surprisingly DOES work(side-effects unknown)
34
Who encrypts email when saving emails?
Who encrypts sent email?
Who signs sent emails?
35
Calendar
Users with wrong(?) calendar settings?
1=Sunday2=Monday3=Tuesday5=Thursday
36
Cache.ndk
Users where Cache= in notes.ini points to– A dedicated file/path– Partly filepaths in which users
might not have write permissions(e.g. Notes program files directory)
37
Clustering / Loadbalancing
FailoverSilent (defaults to 0) is disabled for almost all users:
38
Client Configuration Health
Basic vs. Standard clients
Clients with wrong InstallMode= setting in notes.ini
39
… same for ini:InstallType=
Identifying Client/Admin/Designer configurations:
40
Hardware/OS details: disk space
Users with too little free disk space– might soon call helpdesk– may experience stability issues– have high disk fragmentation = slooooow
41
Hardware/OS details: memory
Users with too little memory– again, sloooow(er) client experience
42
Locations: do not use IP addresses as mailserver names …
A couple of users have an IP address configured as their mailserverbreaks Policies
DNS names as mailservers could become a problem if the DNS domain were ever to be renamed …
43
Mailfile replicas?
98 users work off a local replica+330 managed replica users
BUT: 3,407 users have a local replicaand: 149 users have more than one mail replica …
– 39 of these local replicas are beyond cutoff:
44
ECL
Open doors
45
Analyzing desktop icons (special vs. local vs. server)
46
Analyzing desktop icons (details)
196,930 local databases with an icon (e.g. bookmark.nsf)
380,243 local databases without an icon (e.g. help files, cache.ndk, …)
1,266 templates on desktops
37,108 templates not on desktops (think shared data directory)
36,865 replicas without any icon
267 replicas without a local icon
2,686 replicas without a server icon
862,395 template replicas without any icon
14 template replicas without a local icon
765 template replicas without a server icon47
Summing it up …
48
Conclusions
Mission accomplished– Provided a holistic view across servers and clients
Mission not accomplished (yet)– This is work in process
Nationwide is the 1st customer out of many that leverages most of the features/potentials of the N/D 8.5 platform
– Optimization potential almost exclusively in areas “without features”– Implemented Domino password security the way it should be
49
What we learned
Detailed data helps to leverage IBM Notes and Domino to its fullest potential
… and helps shifting from reactive to proactive
Assumptions vs. Evidence– Eliminate best guess/hope based working
Find out and focus on what really matters
50
Q&A
Thank you for listening!
51
52 © 2013 IBM Corporation
Legal disclaimer© IBM Corporation 2013. All Rights Reserved.
The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.
References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.
Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.
All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer.
Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others.