What’s New in Oracle Data Pump? OOW2017 Whats New Data Pump Author Roy Swonger Subject Oracle database upgrade and migration Keywords Oracle, export, import, …

Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|

What’sNewinOracleDataPump?OracleDatabase12cRelease2

RoyFSwongerVicePresidentOracleDatabaseUpgrade&Utilities

Updated:28-SEP-2017


SafeHarborStatementThefollowingisintendedtooutlineourgeneralproductdirection.Itisintendedforinformationpurposesonly,andmaynotbeincorporatedintoanycontract.Itisnotacommitmenttodeliveranymaterial,code,orfunctionality,andshouldnotberelieduponinmakingpurchasingdecisions.Thedevelopment,release,andtimingofanyfeaturesorfunctionalitydescribedforOracle’sproductsremainsatthesolediscretionofOracle.

What'sNewinOracleDataPump?


What’sNewinDataPump?

ParallelExport/ImportofMetadataSubstitutionVariables&WildcardsREMAP_DIRECTORYLongIdentifiersupportTRUST_EXISTING_TABLE_PARTITIONSValidation&VerificationoptionsOther12.2FeaturesRecapof12.1Features



ParallelMetadataExport:HowitUSEDTOWork

•  StartwithESTIMATEphase– Gathertabledataobjects– Otherworkersremainidleuntildataobjectsaregathered

• Metadataexportedserially• Dataexportedinparallel

Pre-12.2



ParallelMetadataExport:HowitWorksNow

•  StartwithAnalysisstep– Metadataobjectspassedimmediatelytoworkersastheyarefound– E.g.Worker1findsasetofTABLEdefinitions,theyarehandedofftoworker2

•  ESTIMATEphasestillhappens,butmetadatanolongerheldupbyestimate• Notes:

– Worksfordumpfilejobs,andfornetworkjobsifdestinationdatabaseis12.2– TransportableandFullTransportablejobsarenot(yet)parallelformetadata– ESTIMATEphasenowusesSTATISTICSonly– Restartworksasalways

NewFeaturein12.2



ParallelMetadataExport:Logfile•  12.1.0.2 •  12.2.0.1


• Pre-12.2:– Oneworkerperpartition/subpartition– PQusedifpartitionsarelargeenough– Packagebodiesloadedinparallel

• Withpatchforbug22273229– Indexesbuiltinparallel– Constraintscreatedinparallel– Availableasbackportto12.1.0.2,11.2.0.4

•  Startingwith12.2– Addedparallelimportofmostothermetadataobjects

– Someexceptions•  Types(duetoinheritance)•  Schemas•  Proceduralactions

ParallelMetadataImport



ParallelMetadataImport:Internals•  MetadataisexportedinXMLdocuments

–  EachXMLdocumentindumpfilecontainsnobjectsofagiventype

•  XMLdocumentsareallocatedtoworkers1documentatatime

•  Example:161userstoimport–  Usersareexportedwithupto80usersperXMLdocument– WhathappenswithPARALLEL=8?

• Notes:– Worksforconventional(dumpfile)jobs– Not(yet)fortransportablejobsornetworkmode– Restartworkssameasalways– Statuscommandwillshowmultipleworkersonmetadata

XMLDoc1User1User2…User80

XMLDoc2User81User82…User160

XMLDoc1User161

Worker1

Worker2

Worker3

(Workers8-nwouldbeidle)



ParallelMetadataImport:Logfile• ComparisonwithPARALLEL=8for27586objectgrantsandMETRICS=Y

– 12.1.0.2

– 12.2.0.1



Performance:ParallelMetadataImport•  ExamplesfromE-BusinessSuitetestdatabase

ObjectType

Count

11.2.0.4PARALLEL=32

12.1.0.2PARALLEL=8

12.1.0.2PARALLEL=32WithPatch

12.2.0.1PARALLEL=8

12.2.0.1PARALLEL=32

Comments

OBJECT_GRANT(owner)

27586 49 50 51 10 22 Hardconnectforeachgrant

SYNONYM 43254 105 109 111 25 44

TYPE 4364 108 114 119 111 110 Handledbysingleworker

PROCACT_SCHEMA 606 198 216 214 152 175 Handledbysingleworker

TABLE 33164 923 1160 1298 368 248

OBJECT_GRANT(table)

358649 541 543 578 142 157 Hardconnectforeachgrant

INDEX 53190 6721 5770 360 418 272

PACKAGE 53217 424 476 474 114 54

VIEW 34690 538 583 593 151 184

PACKAGE_BODY 52092 1363 1974 1186 1981 959 Alwaysparallelsince11.2

*importtimeinsecondsWhat'sNewinOracleDataPump?


SubstitutionVariablesforDumpfileName•  Substitutionvariablesfordumpfilename:

– Pre-12.2:%Ugeneratesafixed-width2-digitnumber•  e.g.dumpfile=exp%U.dmp

– Newoptionfor12.2expdporimpdp:•  %lor%L:Incrementingnumberfrom01upto2147483646

– Newoptionsin12.2expdponly:•  %dor%D:DayofMonthinDDformat•  %mor%M:NumberofMonthinMMformat•  %yor%Y:YearinYYYYformat•  %tor%T:FulldateinYYYYMMDDformat

$ expdp system/oracle directory=mydir \ filesize=50K dumpfile=exp%T_%L.dmp full=y ... ... ... . . exported "WMSYS"."WM$METADATA_MAP" 0 KB 0 rows Master table "SYSTEM"."SYS_EXPORT_FULL_01" successfully loaded/unloaded ****************************************************************************** Dump file set for SYSTEM.SYS_EXPORT_FULL_01 is: /home/oracle/exp20160917_01.dmp /home/oracle/exp20160917_02.dmp /home/oracle/exp20160917_03.dmp ... ... /home/oracle/exp20160917_67.dmp /home/oracle/exp20160917_68.dmp /home/oracle/exp20160917_69.dmp /home/oracle/exp20160917_70.dmp /home/oracle/exp20160917_71.dmp /home/oracle/exp20160917_72.dmp Job "SYSTEM"."SYS_EXPORT_FULL_01" successfully completed at Sat Sep 17 23:47:31 2016 elapsed 0 00:03:00



WildcardsforTRANSPORT_DATAFILES• Useawildcardin12.2insteadoflistingeveryfile• Pre-12.2:

• NewFeature:wildcards– *(asterisk)matchesmultiplecharacters– ?(questionmark)matchesasinglecharacter

TRANSPORT_DATAFILES=users01.dbf TRANSPORT_DATAFILES=users02.dbf … TRANSPORT_DATAFILES=data1.dbf TRANSPORT_DATAFILES=data2.dbf …

TRANSPORT_DATAFILES=users*.dbf TRANSPORT_DATAFILES=data?.dbf …

$ impdp system/oracle@pdb2 network_link=sourcedb \ version=12 full=y transportable=always metrics=y \ exclude=statistics \ directory=mydir \ logfile=pdb2.log \ transport_datafiles='/u02/oradata/CDB2/pdb2/user*.dbf’



REMAP_DIRECTORY• AppliestoDDLwheredirectoryspecsareused

– E.g.CREATETABLESPACE• Changedirectoryspecwithoutchangingfilenames• UsefulwhenmovingbetweenOSplatforms

– Example:importingdumpfilecreatedonOpenVMSintodatabaseonLinuxREMAP_DIRECTORY="'DB1$:[HRDATA.PAYROLL]':'/db1/hrdata/payroll/'"



LONGIdentifiers• Oracle12.1.0.2:1-30bytes

–  CREATE TABLE abcdefghijklmnopqrstuvwxyzabcd;

• Oracle12.2.0.1:1-128bytes– IfCOMPATIBLE≥12.2.0–  CREATE TABLE abcdefghijklmnopqrstuvwxyz_abcdefghijklmnopqrstuvwxyz_abcdefgh…

–  Databasename:≤8byte–  DiskGroups,PDBs,rollbacksegmentsandtablespacenames≤30bytes

•  https://docs.oracle.com/database/122/SQLRF/Database-Object-Names-and-Qualifiers.htm#SQLRF51129



LongIdentifierSupportinDataPump•  Long(128-byte)identifiersaresupportedin12.2by

– DataPumpexpdp/impdp– SQL*Loader– ORACLE_LOADERaccessdriver– ORACLE_DATAPUMPaccessdriver– ExternalTables

•  Importingdatabasemustsupport128-byteidentifiers– BecarefulwhenexportingwithVERSION=12.1orearlier



TRUST_EXISTING_TABLE_PARTITIONS• Pre-12.2

– Importingintoexistingtablewasdoneserially– DataPumpcouldn’tbesurethatpartitioninginDBmatchedpartitioningindumpfile

• New12.2Parameter:DATA_OPTIONS=TRUST_EXISTING_TABLE_PARTITIONS – Bigperformanceboost– Ifpartitionsdon’tmatch…error:

ORA-31693: Table data object "SH"."SALES_BIG_PT":"SALES_2000" failed to load/unload and is being skipped due to error: ORA-29913: error in executing ODCIEXTTABLEFETCH callout ORA-14401: inserted partition key is outside specified partition



DataValidation&Verification

•  DATA_OPTIONS=VALIDATE_TABLE_DATA – Importonly– Validatesdateandnumberformatsoftabledata– Defaultisnovalidation

•  DATA_OPTIONS=VERIFY_STREAM_FORMAT – Exportonly– Defaultisnoverification

ExtraValidationforThingsThatShouldNeverHappen

Starting "SCOTT"."SYS_EXPORT_TABLE_01": scott/******** tables=t directory=dmpdir dumpfile=t.dmp reuse_dumpfiles=true Processing object type TABLE_EXPORT/TABLE/TABLE_DATA Processing object type TABLE_EXPORT/TABLE/STATISTICS/TABLE_STATISTICS Processing object type TABLE_EXPORT/TABLE/STATISTICS/MARKER Processing object type TABLE_EXPORT/TABLE/TABLE . . exported "SCOTT"."T" 5.570 KB 1 rows ORA-31694: master table "SCOTT"."SYS_EXPORT_TABLE_01" failed to load/unload ORA-02354: error in exporting/importing data ORA-26009: stream verification error: [1], [0], [0], [0]

ORA-02374: conversion error loading table "DPV"."TEST18" ORA-12899: value too large for column C1 (actual: 500, maximum: 498) ORA-02372: data for row: C8 : '



Other12.2Features(1)• Usedirectpathloadinnetworkmode!

– SpecifyACCESS_METHOD=DIRECT_PATHwithNETWORK_LINK=<dblink> – AllowsnetworkimportofLONGandLONG RAW

• DataPumpavailableinInstantClient– ToolspackageforInstantClient– IncludesSQL*Loader,expdp,impdp,exp,imp

• Viewsthatdescribeavailabletransforms– DBMS_METADATA_TRANSFORMS – DBMS_METADATA_TRANSFORM_PARAMS – DBMS_METADATA_PARSE_ITEMS



SQL*PlusHistory• SQL> SET HISTORY ON • SQL> SET HISTORY 1000 •  Example:

– SQL> hist •  Listsallcommandsfromthehistory

– SQL> hist 7 run • Willrunthe7thcommandfromthelist

•  https://docs.oracle.com/database/122/SQPUG/HISTORY.htm#SQPUG-GUID-CCF9B047-3122-4644-BA06-5FA4B5812E9F



Other12.2Features(2)• Newinteractivecommands

– TRACEparametercanbesetforarunningjob•  Noneedtostop/restartjobfortracingtotakeeffect

– STOP_WORKERcommand•  Killanindividualworkeryoubelievetobehungorstuck

– BothwillbedocumentedinMOSnotes

•  Enhancedlogfiles– WhenMETRICS=Y

•  ShowworkerIDforeachitemprocessed•  Showaccessmethodforeachtable

– Includecontentsofparfileinlogfile

18-SEP-16 15:24:30.950: ;;; ********************************************************** 18-SEP-16 15:24:30.951: ;;; Parfile values: 18-SEP-16 15:24:30.953: ;;; parfile: job_name=md_exp_16_12201 18-SEP-16 15:24:30.955: ;;; parfile: reuse_dumpfiles=Y 18-SEP-16 15:24:30.957: ;;; parfile: logtime=all 18-SEP-16 15:24:30.958: ;;; parfile: metrics=Y 18-SEP-16 15:24:30.960: ;;; parfile: parallel=16 18-SEP-16 15:24:30.962: ;;; parfile: full=Y 18-SEP-16 15:24:30.963: ;;; parfile: logfile=md_exp_16_12201.log 18-SEP-16 15:24:30.965: ;;; parfile: dumpfile=md16_12201_%U.dmp 18-SEP-16 15:24:30.966: ;;; parfile: directory=EBSIMP 18-SEP-16 15:24:30.968: ;;; *********************************************************



Recapof12.1Features• VIEWS_AS_TABLESparameter

–  Letsyouexportthecontentsofaviewasatable

• TRANSFORMparameteroptions– TRANSFORM=DISABLE_ARCHIVE_LOGGING:Y

•  Willdisablearchiveloggingduringimportfortablesand/orindexes

– TRANSFORM=LOB_STORAGE:SECUREFILE – TRANSFORM=STORAGE:N – TRANSFORM=TABLE_COMPRESSION:<compression_clause>

•  LOGTIME=[NONE|STATUS|LOGFILE|ALL]parameter– Willwritetimestampsonstatusand/orlogfilemessages



DataPumpBestPractices•  Forfullexports:

– RoleEXP_FULL_DATABASEisrequired •  Forexportconsistencyuse:

– FLASHBACK_SCN=<scn> – FLASHBACK_TIME=SYSTIMESTAMP alternative:

– CONSISTENT=Y [sinceOracle11.2–LegacyInterface]•  ThiswillincreaseUNDOrequirementsforthedurationoftheexport

• Alwayssetparameters:– EXCLUDE=STATISTICS – METRICS=YES



DataPumpBestPractices

• UsePARALLEL=n – Typicallyn=2x<numberofCPUcores>

• Newfeaturein12.2:ParallelExport/ImportofMetadata– 11.2.0.4and12.1.0.2:applypatchforbug22273229toenableparallelimportofconstraintsandindexes

• EXCLUDE=STATISTICSonexport • Newfeaturein12c:TRANSFORM=DISABLE_ARCHIVE_LOGGING:Y

– 12.1.0.2:Applypatchforbug20778442

PerformanceTips



DataPumpBestPracticesNetworkMode

• Directimportviadatabaselink– Parameter:NETWORK_LINK

•  Runonlyimpdponthetargetsystem-noexpdpnecessary•  Nodumpfilewritten,nodiskI/O,nofiletransferneeded

• Performance:Dependsonnetworkbandwidthandtarget'sCPUs• Restrictionsofdatabaselinksapply

– NewFeaturein12.2:SupportforDirectPathLoadoverdblink(includingLONGdata)•  ACCESS_METHOD=DIRECT_PATH

$ expdp ...



DataPumpBestPracticesNetworkMode

• RealWorldCase:KaiserPermanente,Medicare(USA)– impdponNETWORK_LINKwith8vs16CPUcores•  10GBitconnectionleveragedupto8Gbit•  1TBtablecopiedin~15minð4TB/hour

– NetworkbandwidthandCPUbound



DataPumpNewsinOracle12c• TRANSFORMoptiontoenableAdvanced/HCCCompression

–  Example:•  TRANSFORM=TABLE_COMPRESSION:"compress for query high"

– But:Granularityonlyontheentireimport•  Workarounds:

– Precreateobjects•  Downside:Willslowdownimport!!!or:

– PrecreatethetablespacewithCOMPRESSoption•  create tablespace ARCHIGH datafile 'archigh.ora' size 100G default compress for archive high;

– ThenrunDataPumpwithTRANSFORM=TABLE_COMPRESSION:N •  Thiswilldropallembeddedcompressionattributesassociatedwiththetables•  Nowtablespacecompressionoptionwillbeusedforallnewlycreatedtables



ExportingViewsasTables

TABLE1

VIEW1 expdp system/mgr views_as_tables=scott.view1 ...

impdp system/mgr remap_table=view1:scott.table1 ...

TABLE1



ImprovedLogFiles:METRICS=YandLOGTIME=ALL

METRICS=Y(Added11.2)

BasicLogfile

LOGTIME=ALL(Added12.1)



EnhancedCompressionAlgorithm• COMPRESSION_ALGORITHM

– Definesthecompressionalgorithmwhencompressingdumpfiles•  BASIC Thesamealgorithmusedinpreviousversions.Goodcompression,withoutseverelyimpactingonperformance•  LOW: ForusewhenreducedCPUutilizationisapriorityovercompressionratio•  MEDIUM: Recommendedoption.SimilarcharacteristicstoBASIC,butusesadifferentalgorithm•  HIGH: Maximumavailablecompression,butmoreCPUintensive

– Performance:•  Compressionratio•  CPUusage

– RequiresAdvancedCompressionOptionlicense

$ expdp scott/tiger tables=emp directory=mydir dumpfile=emp.dmp logfile=expdp_emp.log compression=all compression_algorithm=medium



EnhancedCompressionAlgorithm• Customerevaluation

– BASICat3.5TB/hour

– MEDIUM at7.0TB/hour

2x



DatabaseUpgradeBlog-Slides


• https://MikeDietrichDE.com/

Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.| What'sNewinOracleDataPump?

Documents

What’s New in Oracle Data Pump? OOW2017 Whats New Data Pump Author Roy Swonger Subject Oracle database upgrade and migration Keywords Oracle, export, import, …