Upload
srivardan
View
24
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Informatica Server Manager
Citation preview
Informatica
Server
Manager
Last Updated : 01/12/00
Center of ExcellenceCenter of ExcellenceData WarehousingData Warehousing
Server Manager ArchitectureServer Manager Architecture
After this session you should be able to describe
How to Configure Server ManagerHow to Configure Server Manager Creating Sessions and BatchesCreating Sessions and Batches Informatica’s session outputInformatica’s session output Difference in Sequential & Concurrent BatchesDifference in Sequential & Concurrent Batches Loading OptionsLoading Options Different Ways to Launch a SessionDifferent Ways to Launch a Session Configure Post- Session EmailConfigure Post- Session Email Monitor Status of Sessions and BatchesMonitor Status of Sessions and Batches How to load Rejected Data with pmrjldrHow to load Rejected Data with pmrjldr How to Recover a Stopped SessionHow to Recover a Stopped Session
Transformation Process
Source DefSource DefMappingMapping
Target Def.Target Def.SessionSession
Source InformationSource InformationTarget InformationTarget Information
MappingMappingSchedulingScheduling
Error handlingError handlingpre/post session scriptspre/post session scripts
Tuning parametersTuning parametersOutput log informationOutput log information
Transformation overridesTransformation overrides
ServerServer
RepositoryRepository
Server Manager
Process Involved
Configure ServerConfigure Server
Create SessionCreate Session
Run SessionRun Session
Monitor SessionMonitor Session
View LogsView Logs
Tune SessionTune Session
Server Configuration
Native database ConnectionNative database Connection– Used to connect to source and target databasesUsed to connect to source and target databases– Connection name, user ID, password, connection string, Connection name, user ID, password, connection string,
database typedatabase type
Register ServerRegister Server– TCP/IP host name / IP address, port # or dotted IPX/SPX TCP/IP host name / IP address, port # or dotted IPX/SPX
addressaddress
Log files pathLog files path– Server path to write session logServer path to write session log– Server path to write session errorsServer path to write session errors
You may also need to :You may also need to :– FTP Properties if Source/target files will be sent to server by FTP Properties if Source/target files will be sent to server by
FTP FTP – Create External Loader Connections like Sybase IQCreate External Loader Connections like Sybase IQ
Server Output
.perf.perf
.control.control
.dat.dat
.Email.Email
.Error Log.Error Log
.Event Log.Event Log
.log.log
.bad.bad
.idx.idx
ServerServerTargetTarget DatabaseDatabase
Session Configuration Wizard
The Session Manager provides a Session Wizard for The Session Manager provides a Session Wizard for easyeasy
Session Configuration.It has four primary pages :Session Configuration.It has four primary pages : GeneralGeneral
– Source SettingSource Setting– Target SettingTarget Setting– Pre and Post Session CommandPre and Post Session Command– Performance and Advance OptionsPerformance and Advance Options
TimeTime Log Files Log Files TransformationTransformation
Session Configuration Wizard Contd.…
Source SettingSource Setting
Session Configuration Wizard Contd.…
Target SettingTarget Setting
Session Configuration Wizard Contd.…
Working With FilesWorking With Files
– Source and Target FilesSource and Target Files
– Using Multiple Source FilesUsing Multiple Source Files
– FTP Source and Target FilesFTP Source and Target Files
– External LoaderExternal Loader
Session Configuration Wizard Contd.…
File Source File Source – Delimited File Delimited File – Fixed WidthFixed Width
Heterogeneous SourceHeterogeneous Source– Two or more tables existing in different DatabaseTwo or more tables existing in different Database– Two or more flat files existing in different SystemTwo or more flat files existing in different System– Two or more ODBC SystemsTwo or more ODBC Systems– Relational and flat file sources in one mappingRelational and flat file sources in one mapping
File Targets (Output Files)File Targets (Output Files)– Delimited FilesDelimited Files– Fixed widthFixed width
External LoaderExternal Loader
Session Configuration Wizard Contd.…
External Loading External Loading
– Increase Session Performance Increase Session Performance
– Informatica Provides two options Informatica Provides two options
• Configure through Session ManagerConfigure through Session Manager
– Oracle , SybaseOracle , Sybase
– Creates control file and starts process to call external loaderCreates control file and starts process to call external loader
• Create control file with Control File WizardCreate control file with Control File Wizard
– Oracle , Sybase, teradataOracle , Sybase, teradata
– Creates control file onlyCreates control file only
Session Configuration Wizard Contd.…
External Loader Process External Loader Process
NT FileNT File
Unix PipeUnix Pipe
.bad file.bad file
External Loader ProcessExternal Loader Process
DB Reject FileDB Reject File
Control FileControl File
Informatica ServerInformatica Server
Transformation - Session Wizard
AttributesAttributes– Override transformation attributes in mappingOverride transformation attributes in mapping
ExamplesExamples– Change source qualifier filter statementChange source qualifier filter statement
– Tune SQL statementsTune SQL statements
– Enable lookup cachingEnable lookup caching
Aggregate BehaviorAggregate Behavior– Specify Incremental aggregationSpecify Incremental aggregation
– Reinitialize CacheReinitialize Cache
Session Commands - Session Wizard
Pre - Session CommandsPre - Session Commands
– Specify operating system commands that need to Specify operating system commands that need to
be run on the server when the session startsbe run on the server when the session starts
– Pre-session shell commands/.bat file should return Pre-session shell commands/.bat file should return
0 for success and non-zero(any positive) for failure0 for success and non-zero(any positive) for failure
– Good for dropping/truncating tables, deleting .bad Good for dropping/truncating tables, deleting .bad
filesfiles
Session Commands - Session Wizard
Post - Session CommandsPost - Session Commands
– Specify any operating system commands that need Specify any operating system commands that need
to be run on the server when the session endsto be run on the server when the session ends
– Good for rebuilding indexes and dropping Good for rebuilding indexes and dropping
temporary tablestemporary tables
– Can specify different scripts for success and failure Can specify different scripts for success and failure
of session. of session.
E-mail Configuration
Used to send e-mail message after a session runUsed to send e-mail message after a session run– VariablesVariables
• %s : session name%s : session name• %l : total rows loaded%l : total rows loaded• %r : total rows rejected%r : total rows rejected• %e : session error status%e : session error status• %t : table details%t : table details• %b : session start time%b : session start time• %g : attaches the session log to the message%g : attaches the session log to the message
– format tagsformat tags• \t : tab\t : tab• \r : new line\r : new line
Dependent on success and failure tabsDependent on success and failure tabs Always check the session log for more messageAlways check the session log for more message
Process for NTProcess for NT– Must use Microsoft Exchange e-mail systemMust use Microsoft Exchange e-mail system– Exchange client software must be installed on the Exchange client software must be installed on the
serverserver– Exchange service must use network securityExchange service must use network security– User starting Power Mart service must be global User starting Power Mart service must be global
user. Determine the power mart startup account by user. Determine the power mart startup account by selectingselecting• Control panel |servicesControl panel |services• Power mart servicesPower mart services
– Create mail box for the user starting the power mart Create mail box for the user starting the power mart serviceservice
– Create profile for user starting the power mart Create profile for user starting the power mart serviceservice
E-mail Configuration Contd..
E-mail Configuration Contd...
The profile must contain only the following servicesThe profile must contain only the following services– MS Exchange ServerMS Exchange Server
– Personal address bookPersonal address book
Add post-session e-mail recipients to the personal Add post-session e-mail recipients to the personal
address book.address book.
In UnixIn Unix– Unix server uses mail to send post-session e-mailUnix server uses mail to send post-session e-mail
– User who start the pmserver process should have the User who start the pmserver process should have the
mail in their PATH.mail in their PATH.
Launching Sessions
Manual StartManual Start– Manually launch a session from within the Server Manually launch a session from within the Server
ManagerManager Session ScheduleSession Schedule
– Schedule a session using business cycle start,stop Schedule a session using business cycle start,stop & repeat internal & repeat internal
BatchingBatching– Use batches to run session concurrently or Use batches to run session concurrently or
SequentiallySequentially Event Based Event Based
– Configure a session to launch based upon the Configure a session to launch based upon the appearance of an indicator fileappearance of an indicator file
Command Line Command Line – Launch a session from the command line promptLaunch a session from the command line prompt
Launching Sessions
Privileges/Permissions needed to run sessions Privileges/Permissions needed to run sessions
– Create Sessions & Batches privilege with read Create Sessions & Batches privilege with read
and execute sessions and execute sessions
– Sessions Operator privilege with read permissionSessions Operator privilege with read permission
– Super user privilegeSuper user privilege
Time - Session Wizard
Informatica provides business
cycle-based start
date,stop and repeat intervals
for running sessions
Scheduling Batches
Nest sessions and batches to run concurrent and Nest sessions and batches to run concurrent and
sequential , or a combination of the twosequential , or a combination of the two
Batch scheduler has same scheduling options as Batch scheduler has same scheduling options as
session wizardsession wizard
Types of batches Types of batches – Concurrent Concurrent
– SequentialSequential
Scheduling Tips
Load order - Load dependent tables firstLoad order - Load dependent tables first
Run Sessions in Parallel Run Sessions in Parallel – Run concurrently in batchesRun concurrently in batches– Effective with SMP systems or parallel databases Effective with SMP systems or parallel databases – Caution not to overload target or hardware platform Caution not to overload target or hardware platform
Load one target in parallelLoad one target in parallel
Load multiple targets in parallelLoad multiple targets in parallel
Event - Based Scheduling
Configure event-based schedulingConfigure event-based scheduling– Create shell script to send indicator file to directory Create shell script to send indicator file to directory
local to the Informatica Serverlocal to the Informatica Server
– Enter path and file name in the advanced propertiesEnter path and file name in the advanced properties
– Schedule the sessionSchedule the session
Session Log ImpactSession Log Impact– The time the file appearsThe time the file appears
– Deletion of the fileDeletion of the file
Monitor Session Monitor Session – When the watcher is in wait, status = file waitWhen the watcher is in wait, status = file wait
PMCMD - Command Line Interface
Can be used with 3rd party scheduling software, Can be used with 3rd party scheduling software,
scripts or program to give more more versatility in scripts or program to give more more versatility in
running sessionsrunning sessions– E.g. CA7, TivoliE.g. CA7, Tivoli
Must have proper Informatica permissions and Must have proper Informatica permissions and
privileges to take below actionsprivileges to take below actions ActionsActions
– Ping ServerPing Server– Start and Stop Sessions and batchesStart and Stop Sessions and batches
Server returns success or failure messagesServer returns success or failure messages Type DEBUG after the command for more feedbackType DEBUG after the command for more feedback
PMCMD - Syntax
Pinging the Informatica ServerPinging the Informatica Server– pmcmd ping {user_name | %user_env_var} {password | pmcmd ping {user_name | %user_env_var} {password |
%password_env_var} {[TCP/IP:][hostname:]portno | %password_env_var} {[TCP/IP:][hostname:]portno | IPX/SPX:ipx/spx_address}IPX/SPX:ipx/spx_address}
Starting and Stopping Sessions and BatchesStarting and Stopping Sessions and Batches– pmcmd {start | stop} {user_name | %user_env_var}{password pmcmd {start | stop} {user_name | %user_env_var}{password
| %password_env_var}{[TCP/IP:][hostname:]portno | | %password_env_var}{[TCP/IP:][hostname:]portno | IPX/SPX:ipx/spx_address} [folder_name:]{session_name | IPX/SPX:ipx/spx_address} [folder_name:]{session_name | batch_name} session_flag [wait_flag]batch_name} session_flag [wait_flag]
Recover SessionsRecover Sessions– pmcmd startrecovery {user_name | %user_env_var}pmcmd startrecovery {user_name | %user_env_var}
{password | %password_env_var} [TCP/IP:][hostname:]portno {password | %password_env_var} [TCP/IP:][hostname:]portno | IPX/SPX:ipx/spx_address}[folder_name:]session_name | IPX/SPX:ipx/spx_address}[folder_name:]session_name wait_flagwait_flag
After the Session Launches...
Poll | Refresh session StatusPoll | Refresh session Status
View Session DetailView Session Detail
View Performance monitorView Performance monitor
View LogsView Logs
Tune the sessionTune the session
Correct Session problemsCorrect Session problems
Monitor Session
Session DetailsSession Details– Select Server Requests | Session DetailsSelect Server Requests | Session Details
– Number of rows loaded/failedNumber of rows loaded/failed
– Read/Write thoroughputRead/Write thoroughput
– Most current Server messageMost current Server message
– Audit trail repository or session logAudit trail repository or session log
Monitor Session
Performance MonitorPerformance Monitor– Select Server Requests | Session Performance Details or Open Select Server Requests | Session Performance Details or Open
file <sessionname>.perffile <sessionname>.perf
– Need to configure session properties to save the performance Need to configure session properties to save the performance
detail countersdetail counters
– Help determine where session performance can be improvedHelp determine where session performance can be improved
Log Files - Session Wizard
Log file path and nameLog file path and nameReject file path and nameReject file path and nameSession log and archive optionsSession log and archive options
Monitor Session
Session LogSession Log– Select Server requests | Session Log or Select view Select Server requests | Session Log or Select view
log file on button bar or Select Open Log File from log file on button bar or Select Open Log File from
Session Details WindowSession Details Window
– Gives complete details on a specific session runGives complete details on a specific session run
Session Sequence of Events
Lock the session and read session properties Lock the session and read session properties Create session log fileCreate session log file Initialize the DTMInitialize the DTM Allocate session shared memory and creates buffersAllocate session shared memory and creates buffers Execute Pre-session scriptsExecute Pre-session scripts Create Reader and Writer process and connect to the Create Reader and Writer process and connect to the
file or database connection.file or database connection. Execute the Stored proceduresExecute the Stored procedures Form SQL insert, delete, update statementsForm SQL insert, delete, update statements Create Index cache and data cache(s)Create Index cache and data cache(s) Parse TransformationsParse Transformations Move DataMove Data
Session Sequence of Events
Execute post Stored Procedures Execute post Stored Procedures
Write historical incremental aggregation data to Write historical incremental aggregation data to
diskdisk
Close the session logClose the session log
Send post-session e-mailSend post-session e-mail
Return memory to systemReturn memory to system
Session Errors
Session might fail becauseSession might fail because– Server cannot allocate enough memoryServer cannot allocate enough memory– Session exceeds maximum number of sessions Session exceeds maximum number of sessions
configuredconfigured– Server cannot cannot obtain execute lockServer cannot cannot obtain execute lock– Server cannot execute post-session commandsServer cannot execute post-session commands– Server encounters database errorServer encounters database error
Circumstances effecting sessionCircumstances effecting session– Server encounters transformation row errors. Server encounters transformation row errors. – Server executes post-session command Server executes post-session command
unsuccessfullyunsuccessfully– Server is unable to send post-session emailServer is unable to send post-session email
Error Handling - Session Wizard
Error Handling Error Handling Perform RecoveryPerform RecoveryOverride tracingOverride tracingStop or Continue on ErrorsStop or Continue on Errors
Reject Files
Created by the Server to store the data rejected Created by the Server to store the data rejected by the writer and/or target databaseby the writer and/or target database
Conditions causing rejected data include Conditions causing rejected data include – Target database constraint violations,null values not Target database constraint violations,null values not
accepted,out of space accepted,out of space – Target table properties ‘Reject truncated/Overflowed Target table properties ‘Reject truncated/Overflowed
Rows Rows Rejected rows are saved in a file <tablename>.badRejected rows are saved in a file <tablename>.bad Rejected rows are prefaced with an indicator Rejected rows are prefaced with an indicator
columncolumn– e.g. 3,D,1,D, , , , , , , , e.g. 3,D,1,D, , , , , , , ,
Reject Indicators Reject Indicators – DData, ata, OOverflow, verflow, NNull or ull or TTruncatedruncated
Reject File Loader ( pmrejldr )
Rename the .Rename the .bad bad file by appending the suffix .file by appending the suffix . in in
to the filename. E.g., <tablename>.bad.into the filename. E.g., <tablename>.bad.in
From the directory where the reject files are From the directory where the reject files are
located, issue commandlocated, issue command– pmrejldr<config_file_name> [folder_name:] < pmrejldr<config_file_name> [folder_name:] <
session_name>session_name>
• Config_file_name> is for UNIX onlyConfig_file_name> is for UNIX only
• Session and older name are case-sensitive Session and older name are case-sensitive
Recovery
Perform Recovery when session stops before Perform Recovery when session stops before
completioncompletion OPB_SRVR_RECOVERY stores the number of OPB_SRVR_RECOVERY stores the number of
records written to the target tablerecords written to the target table Run session with Run session with Perform RecoveryPerform Recovery checked checked
under under Log FilesLog Files tab tab Uncheck Perform Recovery upon completionUncheck Perform Recovery upon completion Source should remain staticSource should remain static Cannot perform recovery if the target has bulk Cannot perform recovery if the target has bulk
inserts enabledinserts enabled
Performance Tuning and Trouble Shooting
You can tune the performance in four general You can tune the performance in four general
area :area :
– SystemSystem
– Source and Target DatabaseSource and Target Database
– SessionSession
– MappingsMappings
Optimizing the System
Accessing Data Over a NetworkAccessing Data Over a Network
– Flat filesFlat files
– Relational DatabaseRelational Database
– Staging AreasStaging Areas
Using Multiple ServersUsing Multiple Servers
Eliminating PagingEliminating Paging
Optimizing the Source and Target Database
Optimizing the Source DatabaseOptimizing the Source Database
– Optimizer HintsOptimizer Hints
– SQL Servers and Joining Large tablesSQL Servers and Joining Large tables
– Sybase and SQL Server Packet SizesSybase and SQL Server Packet Sizes
Optimizing the Target DatabaseOptimizing the Target Database
– Indexes and key ConstraintsIndexes and key Constraints
– Bulk LoadingBulk Loading
Optimizing the Session
Running Parallel SessionRunning Parallel Session
Using Incremental AggregationUsing Incremental Aggregation
Eliminating Transformation ErrorsEliminating Transformation Errors
Tuning Session ParametersTuning Session Parameters
Optimizing the Session Contd....
Tuning Session ParameterTuning Session Parameter
– Shared Memory allocationShared Memory allocation
– Buffer Block SizeBuffer Block Size
– Index Cache SizeIndex Cache Size
– Data Cache SizeData Cache Size
– Commit IntervalCommit Interval
– Decimal ArithmeticDecimal Arithmetic
– Error TracingError Tracing
Questions