Embed Size (px)
DESCRIPTION
r12 Upgrade
Citation preview
R12.1 Upgrade is Easy
That which we persist in doing becomes easier, not that the task itself has become easier, but that our ability to perform it has improved.
Ralph Waldo Emerson (1803 ‐
1882)
Buy the Book
• the little r12.1 upgrade guide
• Step by Step guide to
upgrading 11i to Release 12.1
www.trutek.com
AgendaFunctional UpgradeUpgrade Drivers – Reasons to Upgrade
Cost Savings Hardware and Maintenance
Simplify, Standardize, Shared ServicesKey R12 New FunctionalityUpgrade TeamUpgrade Roles and ResponsibilitiesSteering CommitteeUpgrade Success FactorsCustomizationsGap Analysis
AgendaAlign Business processes with Oracle ApplicationsTesting ‐
Functional and Load
AssessmentsR12.1 Upgrade Categories:
1. Plan to Upgrade2. Prepare to Upgrade3. Perform the Upgrade4. Finish the Upgrade ‐
TestingUpgrade PathsTechnical UpgradeUpgrade by RequestVerification TasksReducing Downtime
Who Cares?
• Managers
• Super Users• DBAs• Developers• Quality Assurance – Testing• Hardware Architects
R12.1.2 Functional Upgrade• The R12 Upgrade is not just a
technical upgrade
• The functional upgrade consists of mapping new business
requirements with new functional features in R12.
Upgrade Drivers
• Compliance
• Cost Savings– Hardware Standardization
• Competitive Advantage
• Integration• New Functionality
Capability Maturity Model
Technology & Process Convergence lead to Simplification, Standardization and
Shared Services
Simplify
Standardize
Shared
ServicesEfficiency drives Cost Savings
Hardware Upgrade Benefits
Simplify and Standardize• Migrate to more cost-effective 64-bit Linux servers• Migrate Application Servers to 64-bit Hardware
• Support more users with more addressable memory• Utilize Virtualization to help standardize• Implement a SAN with block virtualization
capabilities, aka “snapshots”
Application Upgrade Benefits
Simplify and Reduce Maintenance Costs with:• Real Application Testing• Automate Patching & Provisioning• Automate Diagnostics and Tuning• Automate Change and Configuration
Management• Implement Transparent Data Encryption for
better security
Shared Services
• Drive cost savings and increase information quality. • Consolidate non‐revenue generating, administrative tasks in
Shared Service Centers. – Eliminate redundant processes– Utilize self‐service, – Automate processes, – Standardize common business practices reduces costs.
• Helps create a consolidated view of essential global decision‐
making information• Enhances the timeliness and accuracy of data. With
consistent business processes throughout the enterprise,
information can be gathered uniformly, with consistent
quality.
Key R12 Functionality
Advanced
Global
Intercomp
any
AME
Key R12
Factors
MOAC
UMX
SLA
GRC
Integrated FinancialsSubLedger AccountingSubLedger Currency ViewsMulti Org Access ControlUser ManagementApproval Management
Upgrade Team
FIN
Upgrade
Team
HR
CRM
MFG
PA
IT
Dedicated Team Members
The Upgrade Team should
be independent from daily
operations
Team Members are
selected from the business
as determined by the
upgrade project manager
Upgrade Roles & Responsibilities
Project Managers Functional Superusers
DBAs Developers
Testing Manager Functional ManagerCustomization ManagerTechnical Manager
Learn New FeaturesTestingGap AnalysisVerify Upgrade
Gap AnalysisCustomizations:
IdentifyEliminateMigrate to SOA
Identify PatchesUpgrade TechstacksMeasure DowntimeManage Technical
Upgrade Steps
Upgrade Manager Responsibilities
• Manage the following managers:– Development Manager –
Customizations
– DBA Manager – Technical Upgrade
– Systems Manager –
Hardware Upgrade
– Functional Superusers – Develop Process Alignment
– Testing Manager
• Manage the Steering Committee
Steering Committee
Steering Committee Responsibilities:
• Maintain Executive Support
• Gather User Requirements
• Document User Success Factors
• Organize Users to Execute Test Scripts• Prioritize Goals / Issues• Document User’s Issues
• Measure Progress against User Success Factors and Report to the Executive team.
Technical Upgrade Responsibilities DBAs and Developers
• Technical Testing• Improve the Technical Foundation, hardware, software• Understand existing Customizations• Understand how new requirements affect the customizations• Perform the Technical Upgrade• Modify and implement new customizations and extensions• Reduce downtime
Functional Upgrade Responsibilities
• Functional Testing• Understand AS IS Model• Understand TO BE Model• Document the gap between the business process and OA processes• Align the Business and Oracle Applications• Understand existing Customizations• Understand how new requirements affect the customizations• Investigate, Improve and Implement Setups • Train the Users – for example, iProcurement• Reduce downtime• Verify Upgrade
Competing Priorities
• Re‐implement versus Upgrade
• Hardware Migration
• Data Cleanup• Implement New Modules
• Customizations– Eliminate
• Replace with new functionality• Migrate to SOA
Minimize Risk
1.
Testing Proficiency ‐
Test Scripts should be run after each
patch or clone
2.
Enable Fast Backup and Restore with block virtualization or
snapshots
3.
Let the DBAs and Developers practice their piece of the
upgrade in an independent environment
4.
Decouple other projects from the upgrade: new modules, fix
customizations, hardware migration, new business processes
5.
Perform Load Testing if there is any question about capacity
Teamwork
Time&
Money
Skills
UpgradeSuccess
Manager’s View of Upgrade SuccessTime & Money3 months to 1 year$1000 per user + $500
per user (i‐users) with
$150K minimumIncludes CustomizationsDoes not include
implementing new
modules, hardware
migration, etc.
Skills• Augment missing skills
Rent a Consultant• Upgrade Specialists are
worth the money
TeamworkThe upgrade will not
succeed without
TEAMWORK.Communication is
essential.
Budget
Training 10% - 20%Functional Analysis 20% - 30%Technical Upgrade 20% - 30%Customizations 0% - 60%Testing 30% - 40%
Categories and range of the percentage of the total budget
Team Obstacles
• Infighting• Shirking of Responsibilities• Lack of Trust• Critical Skill Gaps
GapAnalysis
Testing
CustomizationsCustomizations• Identify• Document• Eliminate• Migrate to SOA
• Testing is the basis of
success for upgrades
• Testing requires
practice
• Testing requires
Management
• Testing requires the
best and most
knowledgeable users
UpgradeSuccess
Gap Analysis requires:Superusers that
understand the new
features of R12.1 &
understand the
customizations
Critical Success Factors
Customizations• CEMILIs – Software Framework for Extensions,
Customizations, Extensions, Modifications, Internationalizations, Localizations, and
Integration framework.
• 19 classes of extensions
• Oracle’s published guidelines for developing and implementing custom extensions to Oracle
Applications.
CEMLIConfigurations, Extensions, Modifications, Localizations, and
Integrations CEMLI.
The CEMLI Upgrade includes determining technical impact of Oracle E‐
Business Suite Release 12.1 on CEMLIs, upgrading CEMLIs to the new
technology stack, retrofit of CEMLIs for compatibility and usability on
Oracle E‐Business Suite Release 12.1.
• Identify all customizations • Check‐in all customizations into configuration management• Determine customizations that are replaced by new R12.1
functionality
• Re‐Code customizations• Prepare Customization Upgrade Implementation Plan• Customization Configuration Management
Extens ions
Modifica tions
Customizat ions
Customizations• Identify• Document• Eliminate• Migrate to SOA
•Testing is the basis of
success for upgrades•Testing requires
practice•Testing requires
Management•Testing requires the
best and most
knowledgeable users
Extensions
Modifications
Customizations, Extensions, Modifications,
R12.1 Upgrade
Vanilla
• Starts with the presumption that OOTB processes* will work 80+% of the time
• Time is spent focusing on the true gaps (< 20%) requiring customization
Customize
• Significant time spent understanding current process flows – most staff are not good at abstract thinking
• Create a framework for future process flows. Unrealistic “wish lists”
while
developing future process flows, can make the goals unattainable.
• Simplification and Alignment are more difficult without Customizations
Customization Costs
• Customization inflates lifecycle effort (cost) by at least 70%, based on average complexity
assumptions in this model• Actual complexity and required effort (cost)
may be much higher
Cumulative Business Value of “Small” Business Productivity Improvements
•
Even small improvements in business process performance can yield large
business savings over a 5‐year period
•
This example illustrates the 5‐year business value of various levels of
productivity improvement
•
Every 2% improvement – 10 minutes per day of worker productivity –
yields
$17 million in savings over 5 years
Assumptions
Number of affected employees 10,000
Avg. Annual Fully Burdened Cost/Employee $81,000
Est. Annual Employee Cost Inflation 3%
Employee Hours Worked per Year 1,960
Time Frame (Years) 5
Customizations ‐
BPICost ‐
Benefit Analysis for Business Process Improvement
Annual Operation Costs 1,000,0001st Year 2nd Year 3rd Year 4th Year 5th Year
BPI Costs Implement & Upgrade 300,000 150,000
Annual Customization Maintenance Costs 60,000 60,000 60,000 60,000 60,000
10% lower operating costs with BPI ‐100,000 ‐100,000 ‐100,000 ‐100,000 ‐100,000
260,000 ‐40000 ‐40000 ‐40000 110,000 250,000
Annual Operation Costs 1,000,000
1st Year 2nd Year 3rd Year 4th Year 5th Year
BPI Costs Implement & Upgrade 300,000 150,000
Annual Customization Maintenance Costs 60,000 60,000 60,000 60,000 60,000
20% lower operating costs with BPI ‐200,000 ‐200,000 ‐200,000 ‐200,000 ‐200,000
160,000 ‐140000 ‐140000 ‐140000 10,000 ‐250,000
GapAnalysis
Testing
CustomizationsCustomizations• Identify• Document• Eliminate• Migrate to SOA
• Testing is the basis of
success for upgrades
• Testing requires
practice
• Testing requires
Management
• Testing requires the
best and most
knowledgeable users
UpgradeSuccess
Gap Analysis requires:Superusers that
understand the new
features of R12.1 &
understand the
customizations
Critical Success Factors
Upgrade Iterative Cycles
Implementation/ Upgrade Lifecycle
Improve the TechnologyFoundation
Align
Technology
with Business
ProcessesModify or
Create CEMLIs
Improve
Business
ProcessesRealize Cost
Savings
Added Value
35 35
Gap Analysis –
Align Business with Oracle Applications
Business
Process
ApplicationHas my business changed?What are the new business requirements?What is the long term strategy for the business?
How has R12.1 changed?Is there new functionality that could improve the process model?Understand Oracle’s long term strategy?
Integrate business changes with Oracle Applications new functionality
Develop new process models. Retire/modify customizations that are replaced by new functionality
Integrating the Business with Oracle Applications New Functionality is Iterative
Gap AnalysisFunctional Financial R12.1 New Features
• Multiple Organizations• Global Accounting• Financial Enhancements• Release 11i GL SoB• Release 12 Subledgers• Multi‐Org Access Control (MOAC) – Subledger Accounting (SLA) • Subledger Accounting Model• Sub‐ledger Currency Views• Ledger Sets• Integrated Financials• Subledger Architecture• Multiple Reporting Currencies Shared Services• Multi‐Org Access Control (MOAC) • E‐Business Tax• Payments
Gap Analysis
Review Business Process Best Practices for R12.1
Identify business areas that should adopt best practices that may
differ from current business practices
Further investigate the functional gap by examining the following:• Reasons for the change and areas that benefit from new
functionality
• Functionality that is temporarily disabled or has been made
obsolete
• Changes to user interfaces, terminology or concepts, and menu
options
Test Management
The lack of effective testing is the single most frequent cause of failure to “Go Live”. Testing is a dedicated job. Most companies only lend a resource to a test initiative. The Test Manager should be a dedicated resource. The
Test Manager’s tasks:– Mentor the Testers– Stress the importance of testing on the organization– Ensure adequate testing– Monitor and improve the Test Development process– Coordinate testing with the technical and functional staff
Functional
Technical
Load
Technical Testing• Identify Potential Issues• Understand R12 Architecture
Load Testing• Characterize
the Workload
• Identify
Bottlenecks
• Improve
Performance
Functional Testing• Develop Test
Scripts
• Execute Test
Scripts
• Perform Unit and
Integration Tests
• Test
Customizations
• Test 11i rows in
R12. Functional
testing centers on
new R12
transactions.
Make sure
transactions that
were entered in
11i can still be
used in R12.
Testing Methodology
Test EnvironmentsThe following R12.1 test instances may be
required:
• Development instance for the developers
• Test instance for each new module being implemented
• Patch instance for the DBA• Training instance for trainingMore instances can lead to cloning nightmares
Functional TestingFunctional Testing includes:
• Updating existing E‐Business Suite test scripts to reflect valid navigation paths for
Oracle E‐Business Suite Release 12.1
• Develop new Oracle E‐Business Suite Release 12.1 test scripts and incorporate any
customizations.
• Perform regression testing
R12.1 Upgrade Categories
1.
Plan to Upgrade
2.
Prepare to Upgrade
3.
Perform the Upgrade
4.
Finish the Upgrade
Release 12.1 Upgrade Paths
Path A DB 9iR2, 10gR2 Apps 11.5.7 or 11.5.8DB Upgrade & Apps Upgrade need to
be completed during the same downtime window.
Path B If the DB already at 11gR1, Apps 11.5.9.2 or 11.5.10.2
Only upgrade the Apps StackPath C Upgrade the DB & Apps in different
phases
Release 12.1 Upgrade Paths
• If upgrading from a release prior to 11.5.7, the upgrade
path may require an interim upgrade to Release 11.5.10.2. • Because of the significant downtime required to upgrade
from Release 11.0 to Release 12, it may be more feasible to
first upgrade to Release 11.5.10.2 and then some time later
upgrade to Release 12. • This requires the functional users to learn Release
11.5.10.2, and perform all the testing for another upgrade. • The amount of work necessary to perform two rounds of
system acceptance testing may justify another day or two
of downtime, so that the upgrade from Release 11.0 to
Release 12.1 can be completed in one longer period of
downtime.
Upgrade Paths
12.0.4 12.0.612.0
11.5.9.2
11.5.10.2
11.5.7
11.5.8
11.0
12.1.111.5.9.2
11.5.10.2
11.5.111.5.6
11.5.10.2
12.0.6 12.1.1
Release 12.1 Upgrade PathsInitial Release
Interim Release
Final Release
R12 Patch11.0, 11.5.1 ‐
11.5.6
Release 11.5.10 CU2 Release 12.0.0
4440000
11.5.7. 11.5.8, 11.5.9* or 11.5.10*
Release 12.0.0
4440000
11.5.7, 11.5.8, 11.5.9.2, 11.5.10.2
Release 12.0.4
6394500
11.5.9*, 11.5.10*
Release 12.1
6678700
12.1.1
Release 12.1.0.2 7303033* includes CU1 and CU2 (consolidated update)Figure 4 indicates that a direct upgrade path exists from Release 11.5.7 to Release 12.0.0.
Database Upgrade Requirements
Release
Certified Database Versions on Solaris
12.1
10gR2, 11gR1 and the 64 bit versions
12.0
10gR2, 11gR1 and the 64 bit versions
11.5.10.2
10gR2 or 11gR1 and the 64 bit versions
11.5.9.2
10gR2 and the 64 bit versions
11.5.7
8.1.7, 9.0.1 9.2, and 9.2‐64 bit
We can see that there is no certified database version that is
certified with both
Release 11.5.7 and Release 12.1.
Therefore, we can’t do the database upgrade before the
downtime window.
Database Upgrade
• The database installed by the 11.5.10.2 RapidWiz is Version
9.2.0.6. This database version does not support Daylight
Savings Time (DST). Therefore, we have two choices:• Upgrade the database to Version 9.2.0.8, which has
support for DST, and then upgrade to Version 11.1.0.7, or• Upgrade the database to Version 10.2.0.3, using the
database Oracle Home provided with the 12.0.4 EBS install.• Since, we require an interim step, and we already have the
12.0.4 DVDs staged, it is easier to use the 10.2.0.3 Oracle
Home than to download the 9.2.0.8 patch and apply all the
E‐Business Suite‐specific database patches.
StartDB
Upgrade
Upgrade Databasedirectly to 11.1.0.7
Use the OH from the 12.1.1 RapidWiz
install
DB version>= 9.2.0.8 &
DST P4
Yes
No
Post DB Upgrade StepsFor R12.1
Fix Korean lexersRun adgrants.sqlGrant to CTXSYSValidate WorkflowRun AutoConfig
Gather Stats for SYSGrants and Synonyms
Fix Syn for Trade Mgmt
DB UpgradeFinished
This guide assumeswe start with EBS version 11.5.10.2 and 9.2.0.6 of the database.
Apps Versionat Least
11.5.10.2
DB versionat Least9.2.0.6
Yes
Upgrade Database to10.2.0.3
Use the OH fromthe 12.0.4 RapidWiz
We upgrade to 10.2.0.3 as an interim step to 11.1.0.7, because this home comes with the 1204 install,
The OH that comes with the RapidWiz install has all the necessary database patches already applied.
Fix DaylightSaving Time
Patch 4
Upgrade Database to11.1.0.7
Use the OH from the 12.1.1
RapidWiz install
RecommendedOption to
Upgrade DBTo 11.1.0.7
Start Upgrade to 11.5.10.2
NoPerform steps inChapter 1 and
Chapter 2Except DB migration
11.5.9Or
11.5.10
Yes
DB at10.2.0.4 or
11.1.0.7
Yes
No
Perform steps1 and 2In Chapter 3, stop at
Step 3 Migrating the DB
DB at10.2.0.4 or
11.1.0.7
Upgrade DB to10.2.0.4 or 11.1.0.7Use the Rapidwiz
11.1.0.7 Oracle Home
No
Yes AllDB patchesare applied
No
Apply DB Patches
Yes
Required:Convert to
OATM
Finish Chapter 3 Perform the UpgradeApply 12.1.1 patch Finish the Upgrade
Finish Chapter 4Post Upgrade StepsOn-line Help patch
UpgradeFinished
Downtime starts here
Downtime ends here
Recommended:Convert to
OATM
OATMEnabled?
No
Yes
OATMEnabled
NoYes
This guide assumeswe start with 11.5.10.2
Overview of Technical Upgrade
Upgrade Timeframe
• Typically, the upgrade to Release 12.1 from 11.5.10.2 will require a
3 to 4 day weekend for downtime, starting at the close of business
on Wednesday or Thursday, for a 3 or 4 day downtime window.
• The database upgrade generally takes 8 to 12 hours, If the database
upgrade is complete prior to upgrade weekend, it’s possible to do a
2 day applications upgrade from 11.5.10.2.
• The database upgrade can be completed independent of the
Release 12.1 applications upgrade and if possible, should be
completed weeks or months before the Release 12.1.1 Upgrade.
• The Applications portion of the upgrade will take 14 to 32 hours
depending on the speed of the server and the amount of data to
upgrade. Testing will take 8‐12 hours, after the upgrade is
complete.
• Backups can be time consuming and recovery should be tested
before the upgrade weekend.
Upgrade Downtime w DB Upgrade
Clone to
Upgrade
Machine
Thursday Friday Saturday Sunday
8 AM
12 Noon
5 PM
Start
Database
Upgrade
12 Midnight
920610203
Database
Upgrade
1020311107
Database
Upgrade
Start
R12.1
Upgrade
R12.1
Upgrade
Backup
Destructi
ve
Testing
Monday
Go Live!
Restore
from the
Backup
Apply On‐
line Help
Patch
Upgrade Downtime wo DB Upgrade
Clone to
Upgrade
Machine
Friday Saturday Sunday
8 AM
12 Noon
5 PM
12 Midnight
Start R12.1
Upgrade
R12.1
Upgrade
Backup
Destructive
Testing
Monday
Go Live!
Restore
from the
Backup
Apply On‐
line Help
Patch
Upgrade Weekend• Start Upgrade on Friday Afternoon• Clone Upgrade instance from PROD – 2 hours• Perform Category 1 Steps – 4 hours• Perform Category 2 Steps – 6 hours• Perform Category 3 Steps – 18 hours• Perform Category 4 Steps – 8 hours
– Customizations – 4 hours– Setups – 4 hours– iSupplier / SSL – 4 hours
• Perform Category 5 Steps – 4 hours– Developer Testing
• Backup the Upgraded instance – 2 hours• User Acceptance Testing – 8 hours• Restore the Backup if Testing was destructive
Plan to Upgrade
Procure Upgrade HardwareCreate Upgraded Instance for Gap AnalysisIdentify Current Issues
Perform Upgrade Assessments:Functional Customization ArchitectureCapacity
Investigate
Audit
Report
Audit• Identify Current Issues• Document “As Is”
Processes• Document “To Be”
Processes
Report• Define the
Issues
• Recommend
Solutions
• Summarize
Improvements
DetailedAssessment
Investigate• Functional Application
Tuning
• System setup affects
performance
• Improve Functional
Processes
• Review User Defined
Functions
• Review Fast Formulas
Assessment Methodology
Functional Assessment
Audit Understand the System Architecture
Modules in UseModules to be Implemented
Identify Business IssuesIdentify & Prioritize Business Goals
Clearly defined and measurableSet Performance TargetsPrioritize Goals
Functional Assessment ‐
Audit
• Identify the gaps between the current release and the new
R12.1 release to determine functional, platform, network
and operating system gaps. Functional experts are required
to map custom processes from 11i to R12.1. Because of
new functionality in R12.1, existing custom processes may
be replaced by new functionality in R12.1.• Assess the database, reporting and interface transition
issues • Evaluate management controls required: timing, resources
and training issues • Recommend an approach (Upgrade vs. Re‐Implementation)
Plan to Upgrade
• Organize Executive Sponsors and Superusers• Identify and Prioritize Upgrade Drivers:
• Implement functional new features,
• More efficient processing,
• Change Chart of Accounts,
• Consolidation to a single global instance
• Understand the hardware requirements and the upgrade path
R12.1 Upgrade Categories
1.
Plan to Upgrade
2.
Prepare to Upgrade
3.
Perform the Upgrade
4.
Finish the Upgrade
Prepare to Upgrade
Practice Testing – Dedicate Testing Resources
Identify and Document Customizations
Gap Analysis
First, the Superusers need training
When are we Ready?
Your not Ready until your Ready
An upgrade is an iterative process
Are we there yet?
• “A good plan, violently executed today, is better than a perfect plan next week.”
‐
George S. Patton
• We are ready when:– Business processes align with Oracle Applications– Customizations support the business processes
– Users are familiar with the new functionality and have been trained
– Testing indicates all processes are functioning
Optimize•
Reduce Downtime•
Improve Uptime•
Improve Reporting•
Improve Response•
Reduce Costs•
Improve Visibility•
Increase Productivity
Evolve•
Incorporate R12 New Functionality•
Replace customizations with R12
New Functionality
•
Perform Business Process Gap
Analysis
•
Enable EBS Users
Manage•
Align Business Processes with OA R12•
Set Process Expectations•
Provide Training•
Migrate Customizations to SOA•
Implement Comprehensive Business
Reporting
Manage Optimize
Evolve
The Evolution of the R12 Upgrade process It’s an iterative approach
Upgrade Iterations
Rapid Install 12.1 Apps TierTEST
Upgrade
Iterations
1st
Pass R12.1 Apps Tier
Test and Fix
+ + Fixes
R12.1 Apps Tier + 2nd
Pass
Performance
Customizations
+ + New Fixes
R12.1 Apps Tier ++
Customizations
Test
Test
No New Issues
Prepare to Upgrade
Maintenance Wizard (215527.1)– Step‐by‐step, graphical user interface for
performing upgrade tasks
– Consolidates instructions from multiple sources to present a comprehensive upgrade picture
– Reduces upgrade tasks by filtering out those that do not apply to you (using TUMS)
– Indicates critical patches that your system requires
– Can automatically execute upgrade tasks for you
Prepare to Upgrade
If possible complete the following prior to the R12.1 upgrade weekend:
– Upgrade the Database to 11.1.0.7– Migrate to OATM
– Install the R12.1 software– Run Downtime Reducing steps
– Run pre‐upgrade verification steps
R12.1 Upgrade Categories
1.
Plan to Upgrade
2.
Prepare to Upgrade
3.
Perform the Upgrade
4.
Finish the Upgrade
Perform the UpgradePerforming the Upgrade is not iterative
The production upgrade should work predictably
What happens if it fails?Testing, Upgrade checklistsPlan for a secondary upgrade weekend contingency
If the Upgrade fails, you haven’t practiced enough. Look at failed upgrades as forced practice.
That which we persist in doing becomes easier, not that the task
itself has become
easier, but that our ability to perform it has improved.
Ralph Waldo Emerson (1803 ‐
1882)
R12.1 Upgrade Categories
1.
Plan to Upgrade
2.
Prepare to Upgrade
3.
Perform the Upgrade
4.
Finish the Upgrade
Finish the Upgrade
• Perform the Testing
RecommendedOption to
Upgrade DBTo 11.1.0.7
Start Upgrade to 11.5.10.2
NoPerform steps inChapter 1 and
Chapter 2Except DB migration
11.5.9Or
11.5.10
Yes
DB at10.2.0.4 or
11.1.0.7
Yes
No
Perform steps1 and 2In Chapter 3, stop at
Step 3 Migrating the DB
DB at10.2.0.4 or
11.1.0.7
Upgrade DB to10.2.0.4 or 11.1.0.7Use the Rapidwiz
11.1.0.7 Oracle Home
No
Yes AllDB patchesare applied
No
Apply DB Patches
Yes
Required:Convert to
OATM
Finish Chapter 3 Perform the UpgradeApply 12.1.1 patch Finish the Upgrade
Finish Chapter 4Post Upgrade StepsOn-line Help patch
UpgradeFinished
Downtime starts here
Downtime ends here
Recommended:Convert to
OATM
OATMEnabled?
No
Yes
OATMEnabled
NoYes
This guide assumeswe start with 11.5.10.2
Overview of Technical Upgrade
CPCs, UPCs, CPUs, PSUs
• See My Oracle Support Knowledge Document 557869.1, EBS: R12 Oracle Financials Critical Patches
for links to the latest Critical Patch Collections (CPCs) and Upgrade Patch Collections (UPCs).
• A new type of patch collection was introduced in July, 2009 called Patch Set Updates (PSUs). See My
Oracle Support Knowledge Document 854428.1, Intro to Patch Set Updates (PSU)
for more details
about PSUs. • Oracle also releases quarterly Critical Patch Upgrades
(CPUs) that should be applied as they become available.
Critical Patch Collections (CPCs)• Oracle Financials Release 12.0 Critical Patch Collections
(CPC) are consolidated critical patches that all Release 12.0 Oracle Financials customers must
apply to ensure
proper operations of their systems. CPCs are specifically targeted for Oracle Payables and Receivables, and
include dependent fixes for Subledger Accounting, Tax, and Payments. Advantages of applying CPCs over one‐
off fixes and RUPs are as follows:
• CPCs are fully quality assured against current RUP levels. Individual one‐off patches are not.
Critical Patch Collections (CPCs)
• CPCs are consolidated and only contain critical patches that
apply to broad customer usages. They are smaller in footprint
and therefore much easier to apply and uptake than RUPs. • CPC Readmes have detailed business and functional
information about the fixes included. Customers can leverage
the Readmes to determine impact and testing required for
specific process flows and software components involved. • We did not spot any CPCs for Release 12.1 as of November,
2009, but you should continue to check before finalizing your
upgrade. If a CPC becomes available after you complete your
upgrade, you should research the CPC and consider
implementing it at some point to your environment.
Release 12.1 Financials Upgrade Patch Collection (UPC)
The latest Release 12.1 Financials Upgrade Patch Collection
(UPC)
contains Release 12.1 Upgrade patches for the following products: – Payables – Receivables – Tax – Subledger Accounting – Intercompany – Financials For India
The Release 12.1 Financials Upgrade Patch Collection is created
specifically for customers who have yet to upgrade their product
environment to Release 12.1.
This UPC contains improvements to
the upgrade process to Release 12.1.
Release 12.1 Financials Upgrade Patch Collection (UPC)
• These fixes are not required to be applied to environments that have already been upgraded to
Release 12.1.• These patches are critical to the upgrade process and it
is essential that they are applied to your Applications Code Tree (APPL_TOP) prior to running the R12.1
Upgrade Driver. • The latest patch as of November, 2009, is Patch:
8773483:R12
.FIN_PF.B, referenced in My Oracle Support Knowledge Document 880275.1.
• You should look for the latest UPC on My Oracle Support and add it to your upgrade plan.
Critical Patch Updates (CPUs)A Critical Patch Update (CPU) is a collection of patches for multiple
security vulnerabilities. It also includes non‐security fixes that are
required (because of interdependencies) by those security patches.
Critical Patch Updates are cumulative, except as noted, but each
advisory describes only the security fixes added since the previous
Critical Patch Update. Thus, prior Critical Patch Update Advisories
should be reviewed for information regarding earlier accumulated
security fixes.
• Critical Patch Upgrades (CPUs) are released quarterly. When you
upgrade to Release 12.1.1, you should plan to apply the latest
available CPU. The latest as of November, 2009 is the Oct 2009, Rev
1, 20 October 2009. You should look for the latest CPU on My
Oracle Support and add it to your upgrade plan. We recommend
applying the CPU patches on another weekend following the
upgrade. However, if time allows on the R12.1 upgrade weekend, it
is possible to apply the latest CPU
.
Patch Set Updates (PSUs)
• Patch Set Updates (PSUs) are database patches. The PSU stategy is
to deliver low‐risk, high value content that has a limited scope and
is thoroughly tested.
• According to My Oracle Support Knowledge Document 854428.1,
Intro to Patch Set Updates (PSU):
• Patch Set Updates (PSUs) are proactive cumulative patches
containing recommended bug fixes that are released on a regular
and predictable schedule. PSUs are on the same quarterly schedule
as the Critical Patch Updates (CPU), specifically the Tuesday closest
to the 15th of January, April, July, and October.
• The Patch Set Updates for Database 10.2.0.4 and Database 11.1.0.7
each include a Cluster Ready Services (CRS) patch. Like the
Database server patch, the CRS patch is a well‐tested, low‐risk
patch of recommended fixes.
Database PSUs
Oracle Database PSU Unix Patch Comments11.1.0.7.1 Patch 8833297
11.1.0.7.1 for CRS Patch 8287931 Originally released as CRS bundle #1
10.2.0.4.2 Patch 8833280
10.2.0.4.2 for CRS Patch 8705958
PSUs
• The following PSUs are planned for the next Patch Set Update release (January 2010):
– Database 10.2.0.4.3– Database 11.1.0.7.2– CRS 11.1.0.7.2– Enterprise Manager Grid Control ‐
OMS 10.2.0.5.2
– Enterprise Manager Grid Control ‐
Agent 10.2.0.5.2• For the purposes of our Vision instance upgrade,
we need to add either Patch 8833297
for RDBMS Version 11.1.0.7.1, or Patch 8833280
for RDBMS
Version 10.2.0.4.2 to our upgrade plan.
Optimal Number of WorkersIn our upgrade machines there are 4 CPUs. With timing data from more than 30
upgrades, the times were measured and the number of workers was
varied
from 2 to 16. The fastest upgrade times were for 3 workers.
Set the number of workers to CPUs – 1
For a machine with 16 CPUs, then, we would use 15 workers.
My reasoning for this rule is as follows:Patches run jobs that are dependent. When too many jobs are running in
parallel, it is easier to get the jobs out of order, and more jobs will be deferred.
Deferring jobs causes more management overhead.
Therefore, by running one less worker than CPUs, the operating system CPU
requirements and other CPU usage attributed to overhead for deferrals are free
to run on the available CPU and should not disturb the running workers.
Optimal Number of Workers
Oracle Recommends:
To determine optimal number of workers, test with the
following goals:
• Between 1*CPUs and 1.5*CPUs• Average IO response times below 10‐15 milliseconds• Average CPU usage below 100%
Optimize adpatch
• Use TUMS to eliminate the tasks that are not relevant
for your system• Use Shared file system for Multi‐node• Use Distributed AD for Multi‐node• Estimate tablespace sizes for test upgrade using
Note: 399362.1
Choose proper batchsize •
Batchsize=10000
Tuning the Upgrade
time adpatch
defaultsfile=$APPL_TOP/admin/VIS/adalldefaults.txt
logfile=adpatch.log patchtop=$AU_TOP/patch/115/driver
workers=3 interactive=yes options=novalidate,
nocopyportion, nogenerateportion
Other options:
nocompiledb,nocompilejsp,noautoconfig
AutoConfig in Parallel Mode Across Multiple Nodes
The ability to run AutoConfig in parallel across multiple nodes was
introduced in the TXK 12.1.1 Consolidated Patch. This feature reduces
maintenance downtime.
AutoConfig can be run in 'parallel mode' by issuing the following command:On the Applications tier:
perl <AD_TOP>/bin/adconfig.pl contextfile=<CONTEXT_FILE>
[product=<product_top>] –parallelOn the Database tier:
perl <ORACLE_HOME>/appsutil/bin/adconfig.pl
contextfile=<CONTEXT_FILE>
–parallelWhen running AutoConfig simultaneously on multiple nodes, the '‐parallel'
option must
be specified while starting AutoConfig on every
node.
Otherwise, the execution of AutoConfig processes on individual nodes will
not be synchronized, and may result in an inconsistent file system or
database.
AD_TASK_TIMING
JOB_NAME PRODUCT PROGRAM PHASE elapsed time -hrsadobjcmp.sql ad AutoPatch 27 5.652222222adobjcmp.sql ad AutoPatch 79 2.133611111adsstats.sql ad AutoPatch 346 1.908888889adstatrp.sql ad AutoPatch 346 1.908888889adpcpcmp.pls ad AutoUpgrade 12 1.842222222adpcpcmp.pls ad AutoUpgrade 12 1.768611111adobjcmp.sql ad AutoPatch 346 1.758055556adpcpcmp.pls ad AutoUpgrade 12 1.656944444invmenu.ldt inv AutoPatch 41 1.288333333ontmenu.ldt ont AutoPatch 41 0.978888889afoamadmmenu.ldt fnd AutoPatch 41 0.870555556ozfmenu.ldt ozf AutoPatch 41 0.806111111akdatsb1.drv ak AutoUpgrade 20 0.802222222oksmenu.ldt oks AutoPatch 41 0.783055556fndwfusermenu.ldt fnd AutoPatch 41 0.598333333iexmenus.ldt iex AutoPatch 41 0.523055556systemadministratormenu.ldt fnd AutoPatch 41 0.4675fem_xdmi.odf fem AutoPatch 25 0.419166667oamdiagbasemenu.ldt fnd AutoPatch 41 0.419166667bermind.sql ben AutoPatch 2 0.412222222
Tuning the UpgradeMake sure you reset the following init.ora parameters after completion of R12 upgrade
driver
• recyclebin• parallel_max_servers• job_queue_processes
• Merge all the NLS patches and apply them as single merged patch• Isolate post upgrade concurrent programs to a separate manager queue as
mentioned in the best practices Doc id: 399362.1
• Gather statistics before upgrade • Use Gather Auto option at 10g• Record timing for each step during test upgrade• Drop MRC schema before R12 upgrade
Add PL/SQL no compile option in R12 upgrade driver to save time during upgrade– Add “extension plsql_no compile yes”
line in upgrade driver file to enable PL/SQL no
compile option
Reducing Downtime
JOB_NAME PRODUCT PROGRAM PHASE elapsed time -hrs
facpupg.sql FA AutoPatch 245 0.005277778
glrsgup2.sql GL AutoPatch 244 0.010833333
Fixed Assets has a potentially long running sql script, facpupg.sql. The script is run by AutoPatch (adpatch) and takes about 19 seconds to run. The script takes so little time to run that it should be run during the downtime window and would not save a significant amount of downtime if run in advance.
Upgrade by RequestUpgrade the most important data, the last fiscal year
or other periods, during the upgrade downtime and migrate the rest of the historical data later.
The products that use Upgrade by Request are:
• Customer Relationship Management (CRM)• Financials and Procurement• Projects• Supply Chain Management
See My Oracle Support Knowledge Document 604893.1, R12: FAQ for the SLA
Upgrade: SLA Pre‐Upgrade, Post‐Upgrade, and Hot Patch.
Upgrade by Request
• Historical data upgrade depends on product. For some products only SLA data will be upgraded and for
others, both transactions and accounting data will be upgraded.
•
Implementation is a two step process:1.
Set range of periods of the historical data to be
upgraded before R12 Upgrade and run pre‐ upgrade concurrent program
2.
Run SLA post upgrade (upgrade by request option) after R12 upgrade
• Review Appendix G in R12 upgrade manual for more details
Default Upgrade Periods ‐
Minimum Downtime Upgrade
During the upgrade, existing accounting data from the subledgers (AR, AP, GL) is upgraded into the new
Oracle Subledger Accounting (SLA) data model.
By default, the upgrade migrates data for the current fiscal year and the periods of the previous fiscal year
that are necessary to ensure there are at least six periods in the upgrade (if the upgrade is performed in
the first half of the fiscal year).
Install the SLA Pre‐Upgrade Concurrent Program
To change the default number of periods of historic data to be upgraded:
• Apply Patch 5233248
to your Release 11i APPL_TOP• Submit the SLA Pre‐Upgrade Concurrent Program
Run the SLA Pre‐Upgrade Concurrent Program
When submitting this program, enter the following parameters: • Migrate all Sets of Books:
• Yes (SLA Pre‐Upgrade program updates the periods in all Sets of Books)
or • No (SLA Pre‐Upgrade program updates the periods that belong to the
selected Set of Books).• Set of Books: Set of Books to be upgraded where you have selected to
upgrade one Set of Books.
• Start Date: Date used to determine the first period to be upgraded. The
first period is the first period the Start Date falls in; it does not have to be
the starting date of the first period.
The start date can be changed to a date earlier than the 6 months
minimum, but not shortened to less than the default.
Run the SLA Pre‐Upgrade Concurrent Program
You may need to run the SLA Pre‐Upgrade program if you are
using Oracle General Ledger and at least one of the following
subledgers:•
Assets
•
Cost Management
•
Payables
•
Receivables
•
Purchasing
•
Project Accounting
This optional program allows you to change the default
number of periods of historic data to be upgraded.
Run the SLA Pre‐Upgrade Concurrent Program
The Release 12 SLA Pre‐Upgrade program considers the
following modules:
• Payables • Receivables • Purchasing • Project Accounting • Inventory/Costing • Fixed Assets (Fixed Assets uses GL periods)
SLA Pre‐Upgrade programThe SLA Pre‐Upgrade program uses the GL_PERIOD_STATUSES
table. The
MIGRATION_STATUS_CODE column indicates the
status of the record in the GL_PERIOD_STATUSES table:
P = Pending Upgrade ‐
The system is preparing to upgrade the
accounting periods in this table that have a status P. The
accounting transactions in these corresponding accounting
periods will be upgraded from Release 11i to Release 12 by the
Subledger
products (i.e, AR, AP etc).
U = Upgraded ‐
The accounting records have been upgraded from
Release 11i to Release 12 N = New ‐
Only used by GL
<blank> = These records do not meet any of the criterion
mentioned and will therefore not be considered in the upgrade.
SLA Pre‐Upgrade Program
The parameters for the SLA Pre‐Upgrade program are: • Upgrade all Sets of Books, or • The specific Set of Books and the Start Date.
The Release 12 SLA Pre‐Upgrade Program tags the accounting
periods in the GL_PERIOD_STATUSES table with P for pending
upgrade when the records are for AP, AR, Project Accounting, FA,
Inventory/Costing and PO products only.
No other products are considered by this pre‐upgrade program
Adjustment Periods
• The accounting period under consideration must be a NON‐
Adjustment period.• Adjustment periods are not upgraded because the upgrade that
was designed by the Subledger
Product teams (AP, AR etc) caters to
the transactions created by those products and posted to GL. • Typically, the Subledger
Product teams do not generate
transactions that post into adjusting periods. Given that adjusting
periods overlap dates with non‐adjusting periods, the logic for
mapping a transaction into an accounting period is to take the GL
date and figure out which non‐adjusting period it falls into.
The period has a status of closed, open, future, and never opened. • Note: Never
opened is only applicable to the Inventory/Costing
product.
SLA Post Upgrade Processing
Upgrade by Request If you do not perform a complete upgrade of the accounting
data, Oracle Subledger
Accounting allows you to perform an
additional upgrade of the data by running the SLA post‐
upgrade process Concurrent Program whenever the missing
data is required.
Global Accounting Engine Verification Tasks
Run Accounting ReportsYou should run the Global Accounting Engine accounting reports before the upgrade and the corresponding Subledger
accounting reports after the upgrade to ensure that you have a proper audit trail of the upgraded accounting data. The reports are as follows:
Global Accounting Engine
Subledger
Accounting
Daily Journal Book
Daily Journal Report
Account Ledger by Account
Account
Analysis Report
Supplier and Customer Subledger
Third Party Balances Summaryby Account
Supplier and Customer Balance
Third Party Detail and Balances by Account Report
E‐Business Verification Tasks
E‐Business Tax Verification Tasks
Tax Transaction Audit and Reconciliation ReportsTo ensure that transaction tax information has been correctly upgraded, run
the Payables Tax Audit Trail report and the Receivables Tax Reconciliation
report for the current tax period before the upgrade to set a benchmark of
transaction information. Then immediately after the upgrade, run
the same
reports in the new environment for the same period and compare the
results to ensure that the tax values are still the same.Payables and Receivables Transaction QueryFor a sample of Payables and Receivables transactions, record the details of
the associated tax for these transactions before migration, and then query
them again after the upgrade to ensure that the tax has been correctly
upgraded. Duplicate the same transactions and re‐trigger to ensure that the
new E‐Business Tax‐based calculation is consistent with the previous
calculation.
Payments Verification Tasks
Legal Entity Configurator
Verification Tasks
Legal Entities and EstablishmentsYou should perform a review of all legal entities and establishments in your
system after the upgrade is complete to ensure that the correct legal
structure is in place. You can access this information by using the Search
Page in the Legal Entity Configurator.
Refer to the Oracle Financials and Oracle Procurement Functional Upgrade
Guide: Release 11i to Release 12.
If you need to create or upgrade legal entities and establishments, then see
the Oracle Financials Implementation Guide
for instructions.
Payments Verification TasksPayables Verification TasksTrial Balance Reconciliation•
In your Release 11i
environment, run the Accounts Payable
Trial Balance, Posted Invoice Register, and Posted Payment
Register reports. After the upgrade, run the Open Account
Balances Listing Report, Posted Invoice Register, and Posted
Payment Register in your upgraded environment and
compare the results.•
The reports run for a ledger or a ledger set, not within the
context of a single operating unit. The Release 11i Trial
Balance and Posted Invoice and Payment Registers run
within a single operating unit. Depending on your system
configuration, you may need to sum several of the 11i
reports to tie to the new versions.
Payables Verification Tasks
Payables Verification TasksInvoice and Payment ProcessingTo verify the integration with Oracle Payments and the upgrade
of existing invoices, submit a payment batch with limited
selection criteria in order to pay a few invoices.
Accounting Setup and ProcessingQuery an invoice that was not validated prior to the upgrade,
then submit accounting for that invoice. Query an invoice that
was accounted before the upgrade, cancel it, pay it, and then
account for the payment. Also see Global Accounting Engine.
Payments Verification Tasks
Payments Verification TasksThese tasks apply only to Oracle Payments. In general, your
planning for upgrade verification should involve testing in the
two payment process areas:
Funds Disbursement
‐
If you used Oracle Payables for issuing
payments in Release 11i, you should plan to test the funds
disbursement processes equivalent to the former payment
batch flow to ensure that your upgraded data correctly reflects
your business process.
Payments Verification Tasks
Payments Verification Tasks
Funds Capture
‐
If you used Oracle Receivables for electronic
payment processing such as direct debits or bills receivable
remittances, you should plan on testing these areas to ensure
that your upgraded data correctly reflects your business
process.
If you used Oracle iPayment
for capture of funds from credit
cards or bank account debits, you should plan on testing these
processes to ensure that the upgraded data results in the
process you expect.
Payments Verification Tasks
Payments Verification TasksSystem Security OptionsOracle Payments provides this new page where system‐level
settings for encryption, masking, and credit card security can be
controlled. When your upgrade is complete, you should plan on
reviewing the seeded settings in this page to ensure they meet
your business needs.
For example, in Release 11i
masking of credit card values is
controlled in different ways throughout the applications. In this
release, the central setting in this page controls all masking. You
will want to review the setting in this page and modify it if
needed.
Payments Verification Tasks
Payments Verification TasksOracle Payables ImpactYou may want to run reports for use in your upgrade
verification testing.
For example, you may want to use the Suppliers Report in
Oracle Payables to verify the data upgrade for payment details
and bank accounts on the payees created in Oracle Payments.
You can use any reports that you ran before the upgrade to help
verify upgraded data.
Payments Verification Tasks
Payments Verification TasksIn addition, there are some key setup entities that should be
reviewed and used in testing payment processing.
Payment Process Profiles ‐
You should plan on reviewing the
settings for the seeded profiles created by the upgrade. These
settings come from a variety of sources, and since the profile
drives the entire funds disbursement flow it is important to
verify that the setup supports your business process. You should
pay special attention to the usage rules set on the seeded
profiles as these can be changed if the upgraded values do not
align with your needs. It is recommended that you run a test
payment process with each profile that you plan to use in
production.
Payments Verification Tasks
Payments Verification Tasks
Payment Methods ‐
A new setup page is provided where
payment methods can be created or updated. You should plan
on reviewing the payment methods seeded by Oracle Payments
to ensure that they meet your business needs.
Payment Systems and Accounts ‐
You should plan to verify
these entities after the upgrade, and in particular the required
settings, values, and their links to the payment process profiles.
This setup controls important parts of the funds disbursement
process such as payment file transmission, so you should test
this area to be sure that the process is working as you expect.
Questions
• Thanks
• The paper / presentation is available at:www.trutek.com
Shared Services
• Services can be shared at many different levels, and shared service centers can exist for different
reasons. – Order desks, – Reporting Centers, – General Ledger Centers, – Disbursement Centers, – Inventory Management Centers,– Procurement Centers.
• Many of these centers may be combined as one center.
Integrated Financials
• The new bank account and disbursement models facilitate the payment of
invoices and other payables out of different operating units, from an
appropriate bank account, and with the appropriate intercompany
handling.
• Intercompany processing is dramatically revised by enhancements in both
Financial’s
intercompany management and Supply Chain Management’s
Enhanced Drop Shipments. – Rather than a GL only system, Financials now links into Receivables and
Payables to generate matching and tied documents (configurable though Bill
Presentment) and a new reconciliation scheme.
• Related SCM products provide transfer pricing modeling and
enforcement, inventory consignment (at subsidiaries or otherwise), and
tracking of profit in inventory.
• All feedback to General ledger and the Financial Consolidation Hub for
elimination.
Subledger
Architecture
• A Set of Books (11i) becomes a Ledger with its own Ledger Set, in Release 12.
• SLA will return the same accounting as the earlier accounting engine did. Operating Units will still
‘stripe’
your transaction data. • Benefits:
– subledger
accounting, – XML publishing applied to reports, – additional DBI portlets
and pages,
– AR‐AP netting, and– gross margin analytics in AR.
Subledger
Currency Views
• Accounting for subledger
transactions at the event in a standard manner
with a single accounting engine allows us to provide multiple accounting
representations for a single event.
• A purchase can be simultaneously accounted for an increment to
inventory for your US GAAP or IAS/IFRS accounting, and as a debit to the
P&L for your national compliance accounting.
• The accounting entity involved can maintain multiple ledgers, each
complying with a different set of accounting principles – we call them
‘accounting methods’, and, of course, the transactions and ledgers can be
valued and denominated in different currencies;
• You can now generate currency views of a ledger at the subledger
transaction, general ledger transaction, general ledger balance,
or
consolidation workplace levels.
Multi‐Org Access Control
• Multi‐Org Access Control enables companies with a Shared
Services operating model to efficiently process business
transactions by allowing them to access, process and report
on data for an unlimited number of operating units within a
single applications responsibility.• This increases the productivity of Shared Service Centers, as
users and processes no longer have to switch applications
responsibilities when processing transactions for multiple
operating units at a time. • Data security and access privileges are still maintained using
security profiles that now support a list of operating units.
Team Obstacles
• Infighting: Effective teams don't have to be made up of people who like each other but there must be
respect for each other– misdirected energy to bickering and undermining
colleagues – members must be willing to set aside petty differences
• Shirking of Responsibilities
When members avoid taking responsibility for both process or running of a
group and for specific assignments, a team becomes a "pseudo team"; i.e., team in name but consistently underperforming.
Team Obstacles
• Lack of Trust
If trust is lacking, members are unable to depend on each other.
• Critical Skill Gaps
When skills are lacking, teams flounder, members have trouble
communicating with each other, destructive conflicts result, decisions aren't made, and
technical problems overcome the group
Teamwork
• Teamwork
is the concept
of people working together cooperatively, as in a sports
team.
• Many large, ambitious projects usually require that people work together, so teamwork has become an important concept in organizations.
• Industry has seen increasing efforts through training and cross‐training to help people to work together
more effectively and to accomplish shared goals, whether colleagues are present or absent.
Team Development
• As teams grow larger, the skills and methods that people require grow as more ideas are expressed freely.
• The intimacy of a small group is lost, and the opportunity for misinformation and disruptive
rumors grows.
• Specifically, leaders might encounter difficulties based on Daglow's
Law of Team Dynamics: "Small
teams are informed. Big teams infer."
