Power Tools for FileNet P8 Content Management Systems · Services 5.x systems to IBM-FileNet P8 enterprise content management repositories. The program migrates metadata, documents

www.BowenECMSolutions.com [email protected]

2010

CS2P8 Migration Program User Manual and Guide By Jay Bowen

mailto:[email protected]?subject=Panagon%20CS%20to%20P8%20Migration�

www.BowenECMSolutions.com || Contact: [email protected] CS2P8: Panagon to P8 Migration Software January 2010

2 CS2P8 Introduction | BowenECMSolutions.com

Contents CS2P8 Introduction .......................................................................................................................... 3

Utilities and Application Detail ........................................................................................................ 4

Data Format (repair, substitute, split, replace) ....................................................................... 4

Data Analytics Feature............................................................................................................. 5

Migration Job Control and Change Management ................................................................... 5

CS2P8 – Installation & Configuration .............................................................................................. 6

System Compatibility and Known Issues ................................................................................. 6

Prerequisites ............................................................................................................................ 7

Installation Instructions ........................................................................................................... 7

Migration Suggestions ..................................................................................................................... 7

Recommended database tuning (pre – run task) .................................................................... 8

Common User Errors ............................................................................................................... 8

Step 1 - Add CS Library .................................................................................................................. 10

Step 2 - Launch CS Migrate Documents ........................................................................................ 13

Step 3 - Migrate CS Folders (Optional) .......................................................................................... 20

Step 4 - Folder via the User Interface (Optional) .......................................................................... 25

Step 5 - Migrate Annotations (Optional) ....................................................................................... 31

http://www.bowenecmsolutions.com/�

mailto:[email protected]�


3 CS2P8 Introduction | BowenECMSolutions.com

CS2P8 Introduction The BowenECMSolutions CS2P8 program supports migrations from Panagon Content Services 5.x systems to IBM-FileNet P8 enterprise content management repositories. The program migrates metadata, documents and folders to the target P8 system quickly and easily. In fact, from installation to migration typically takes 15-20 minutes for novice users. Unlike other migration programs, the CS2P8 program does not require you to stage data, take your CS system offline or require the business unit to work in both systems while data is being migrated. Documents can be migrated in real time without impact to the CS system or P8, trailing documents can be added post migration and the user may selectively migrate current version, all versions or filter migration data per class. All migration job data is collected as XML which is called a 'batch'. The XML contains the metadata for the CS documents and tracks the Panagon Document ID to the P8 ID for instant reconciliation of data. The XML 'batch' is used to promote migration jobs from development to production quickly and easily. Batches as large as 500,000 documents are supported. Multiple computers can concurrently migrate partial batches when a given document class is over 500,000 documents. The CS2P8 program also allows advanced data format functions to standardize metadata fields, obtain data lengths, validate data types, obtain distinct document types, conditionally substitute and split metadata. In addition to document functions, the program provides foldering functions to recreate your entire Panagon folder structure in P8 and file documents appropriately. Features Matrix • Simple to use, no third party COM

DLLs or dependency on IDM Desktop • High speed migration • Extensive logging • Data format functions • Compound document support • Folder Migration and filing • Migrate current version or all versions • Update CS links to P8 links • Site licensed

• Instantly reconcile migrated data • Batch job control • Multithreaded job control with pause

and resume capability • Property map capability to dynamically

renamed CS properties to P8 properties • Include/Exclude filters on metadata

such as starts with, ends with, contains, regular expression

• Migrate at a fraction of what vendors would charge

For a detailed explanation of features and configuration please refer to the detailed instructions that follow.




4 Utilities and Application Detail | BowenECMSolutions.com

Utilities and Application Detail

Data Format (repair, substitute, split, replace) A common requirement during repository migrations is the need to validate, update and translate values. For example, converting old document codes to new document codes would be a common client request when undergoing a conversion. The customer typically believes moving from the old system to the new system is a good time to clean up their data and IT generally feels that what already started as a large migration project is getting even bigger with data value conversions. The CS2P8 program handles these property updates easily and can conditionally substitute or convert data on the fly. The CS2P8 application dynamically creates an XML representation of data to migrate once the user selects the document class node in the tree view display. All manipulation of data is handled on the client side, no CS updates are made. The user then formats data as needed with the in memory data and once satisfied with the changes may save the XML to a file or start the job. As a note to customers migrating from case insensitive CS libraries to case sensitive P8 object stores is the requirement for CVL lists to identically match your doc type code values. While you can force case insensitive searches CVL lists must match the case found in documents or users will not be able to locate documents. The following functions are provided to assist the user in altering document metadata. In all cases the user selects the FROM property and the TO property which do not have to be the same property. This is intended to provide the greatest degree of flexibility to manipulate data:

• Conditional Replace – handles the use of a simple tab delimited text file of old values to new values. Map files of 10,000+ old and new values per map file is easily handled.

• Split and/or Replace – Assume you have a multi-value property called FRUITS

with values Apple, Banana and Orange present. The customer desires to split all three values or just one value into another property, for our example we will split Banana into a new property and rename to Tropical Fruit – Banana. By using a map file with old value “Banana” and new value “Tropical Fruit – Banana” then selecting the property to split “from” and the “TO” field, you can split multi-value properties easily and in real time.

• Rollback – allows most property value changes made to be rolled back with a

click

• Insert Default Value – Inserts a default value on user selectable null or non-null columns




5 Utilities and Application Detail | BowenECMSolutions.com

• Substring – Allows trimming or extraction of values to the same property or a different property.

• Filter – Include or Exclude data based on a property value. This allows the user

to selectively target data that needs concentrated attention.

• Merge, ToUpper,ToLower, Proper, Clone, Pattern Replace, Substring, Substitute, Default value

Data Analytics Feature Another typical requirement is the need to evaluate usage on property fields that store metadata. What sounded good ten years ago for properties may have been a good idea back then but never utilized in production. Unused properties should not be carried forward into P8 or perhaps the property should be a hidden P8 property for randomly populated fields. The CS2P8 program provides analytic tools to cube data (grouping and hierarchical viewing) of CS metadata as well as reporting on number of “nulls” per property within seconds. The included analytic tools provide the application user with 10,000 foot visibility of data to better understand what properties are used, misused and what will need to be repaired prior to moving forward with the migration of the document class.

• Metadata Report – Quickly generate null and non null counts for all properties for the class currently being migrated. Benchmark is < 5 seconds on a batch of 500,000 records.

• Generate Distinct Values – Greatly simplifies the creation of choice lists to a

displayed temp file that can be used for choice list creation. The user may also simply drag the column to the header grid area which will display a grouped by aggregate view of data with expandable rows.

• Data Validation – With a single click validate all rows of the present batch job

conform to property integer or date data types. • Multiple Group By – Allows the user to drag columns into header area and

group by as many columns as necessary

Migration Job Control and Change Management Perhaps the most import tool for data migrations is the need for logging and job control. Migrations can be handled at the document class level and using the same “batch” XML file validate the migration to development, QA and then production. Promoting the migration job between environments is as simple as making the XML batch available to the QA or production environment migration user.




6 CS2P8 – Installation & Configuration | BowenECMSolutions.com

The CS2P8 Application creates an XML file for each batch, which is typically an entire Content Services document class. For large document classes the customer may split the batch where they desire and run the migration application on multiple machines. Upon selecting the Content Services class to migrate the application loads metadata from CS to the machine running the migration program. At this point the user may elect to save the metadata as XML to the local file system. Upon saving the file, the application will automatically name the XML file with the name of the document class being migrated. The application user may select a starting and ending document ID, maximum batch size (100k-500k) or apply a filter to obtain only certain document types like Word, TIF or Drawings. The CS2P8 program will automatically create a log file for each batch as soon as the job starts. The application is multithreaded which shows the P8 GUID as the documents are committed to P8. The job may be paused and restarted, with each document having a GUID being ignored allowing job resumption at the point of which it was last stopped. The log file records the PanagonID and the P8 GUID as well as error messages if any. Lastly, the batch XML file captures that was used for the job records the P8 GUID and minimizes the need for end of batch reconciliation. Job Control Features

• A batch is an XML file of CS metadata that can be saved to the local file system and promoted between environments

• Job start, restart and logging. A progress bar shows current status. • User selectable batch size, starting and ending document ID, user selectable file

type (TIF, Word, Drawing)

CS2P8 – Installation & Configuration

System Compatibility and Known Issues 1. Do not install the program onto host computers that have the FileNet P8 .Net

SDK installed as a registered program. The installed SDK .Net API version will override the version expected by the program and could result in a logon error.

2. VM hosts may not register log file paths correctly for drives other than C:\, this only applies to log file output and working directory for the application. For instance, in the configuration file you enter E:\MyMigration the CS2P8 program may still attempt to write to C:\. This only occurs on the VM host performing the migration.

3. When migrating large documents in excess of 250MB each ensure you have tuned you CE application server appropriately as listed in FileNet documentation. Also review applicable APARS that identify known issues and the appropriate hotfixes.




7 Migration Suggestions | BowenECMSolutions.com

Prerequisites 1. All Microsoft based operating systems are supported. 2. .Net framework 2.0 or better present. 3. Microsoft Web services extensions 3.0 (WSE 3.0), required by FileNet P8 4. Network connectivity to P8 system, gigabit connection highly recommended for

fastest possible performance. 5. Administrative rights to install the program. 6. 2.4Ghz CPU or faster (recommended) 7. 2 gigabytes of memory (minimum) 8. If using Oracle, install oracle ODP client access components.

Installation Instructions 1. Install the software via URL link provided to you. 2. If present, uninstall the existing version using Add/Remove programs. 3. Accept all defaults on installation or optionally choose installation directory of

your choice. 4. Open the installed location folder, enter your license key provided to you in the

license key setting. All settings are contained in the %appname%.exe.config file. Use notepad to edit the file and save. If your operating system is enforcing User Account Control (UAC) you may have to save to your desktop then copy the file back to the installed directory.

5. Enter your GMT offset in the configuration file. If you are an offshore development team consider your local time may be different than the time for the designated system. To easily calculated the correct time, perform a test migration and calculate the time difference as a positive or negative integer in hours then enter the value in the configuration file.

6. The desktop Installation complete, proceed to configuration.

Migration Suggestions

1. In all cases, migrate documents first. Then perform post migration tasks such as

filing, foldering, PDF conversion, reporting, etc.

2. Disable CBR indexing for the given document class during bulk migrations.

3. If you are migrating from MS SQL to Oracle your CS system is likely set up with case insensitive collation and the target P8 system is case sensitive. Make sure you standardize fields when assigned to a choice list.

4. Do not impose data constraints such as required fields or choice lists prior to a class migration. The program will detect most if not all errors but you do not want to interrupt your migration because of a P8 required property does not have a value from CS. Consider that properties can be set to required or not required





as business needs change and data only complies to those rules at the given time.

5. The XML job contains the Panagon ID and the P8 ID for instance reconciliation. If desired query your object store database for the P8 ID and Panagon ID on the docversion table.

6. Use the XML batch jobs and save copies for each operation. Edit the XML with textpad or notepad when you need to tweak items or substitute strings quickly.

7. Optionally create an Access MDB to contain the XML imports.

8. Run the migration program is close proximity to your CS and P8 system to minimize the data payload over the network.

9. Benchmark your migrations, a single workstation can migrate 140 documents per minute, two workstations combined 240 and three stations approximately 300. Results vary based on capacity of P8 system, document size and number of properties.

10. Disable workflow subscriptions on the target class.

Recommended database tuning (pre – run task) 1. You can copy your library server database to another server to minimize impacts

to clients although not necessary. Create the following non clustered database indexes (replace DBO with your schema)

a. on dbo.element.e_ina01 and on dbo.version.v_e_ina01 b. on dbo.element.e_name and on dbo.version.v_e_name c. If you plan on using the doc type filter, create an index on

dbo.version.v_file_name

Common User Errors 1. Do not assume your CS choice list (aka value list) contains all the values of your

documents. A choice list can be added or removed at any time and an administrator may have removed items that once existed. The correct source for value items is obtained from your document data, not your CS choice list.

2. Verify you have correctly set your default storage policy and default instance security before migrating data. Create and use seven default groups that map to the P8 API enumeration for security which are: full control, major version, minor version, modify properties, view properties, view and publish. Name the AD group with the objectstore - class name - % right %, for example gs-FileNetHR-PersonnelRecords-FullControl.





3. Remember that data can be migrated from CS by class and additionally filtered on metadata fields. Consider the impact of creating compound documents if you have partially migrated data. Compound documents are not bound by class.

4. Trailing documents from CS are appended to versions in P8. If you are using the trailing document feature a lookup will be performed to your destination P8 object store to obtain the current version of the document. If you have migrated three versions of a document from CS to P8 then a user creates version 4, 5 and 6 in P8 you will still only have three version documents in CS. If another user in the business unit creates version 4 in CS and you migrate the document the new version number in P8 will be Seven. For example: The p8 current version (v6) + new document from CS (v4)= major version number v7 (p8 v6+ CS v1).




10 Step 1 - Add CS Library | BowenECMSolutions.com

Step 1 - Add CS Library You must first add - register your CS library to the migration program. 1. From the menu, click

configure >> CS Libraries. All other menu items will remain disabled until a library is added.





2. The library manager screen will appear.

3. Enter your server name, library name (database name), user name and password.

4. Click Add library which will launch analysis code against the CS system.





5. At the completion of the analysis operation you can click the library node and LIBRARY VIEW tab to view statistical information for the CS system.

6. The CS Library is now registered with the migration program, initial set up is completed.




13 0BStep 2 - Launch CS Migrate Documents | BowenECMSolutions.com

Step 2 - Launch CS Migrate Documents The CS Migrate documents screen is the main CS to P8 migration program menu. 1. From the menu, click File

>> CS Migrate Documents.





2. Click the P8 Settings Tab.

3. Enter your User Name (windows logon name).

4. Enter your password.

5. Enter the P8 URI / URL.

6. Click TEST URI to test the path to P8.

7. Click LOGON to connect to P8 and retrieve a list of object stores.

8. ccc 9.





10. Now select an object store and document class.

11. 12.





11. From the CS Libraries tab, select a target document class.

12. The current version document count is show alongside the class.

13. The grid will fill with data from your CS system for this document class

14. Ensure your P8 class contains the string property 'PanagonDocID' which is required when migrating data. The property can later be used to reconcile data migrated to the P8 database and will provide a permanent, corresponding reference from Panagon Doc ID to P8 ID.





15. Optional Settings, click the MIGRATION Settings tab. Ensure all options are set before selecting data from the Panagon CS System.

a. Check Current versions only to return the current version only

b. Use the max documents drop down to control a range of documents for classes over 500,000 or if you want to break up smaller classes to multiple files.

c. Enter a starting Panagon document ID to start a job at a specific document ID.

d. Enter an ending Panagon document ID to specify the ending range for a batch job.

e. Use the file type filter drop down to selectively filter documents by extension.





16. Click a CS property then right click to rename it to a P8 property.

a. The context menu will open displaying a list of properties that are available.

b. Click the appropriate property of your choosing from the drop down list of properties for this class.

c. Delete all properties that will not migrate from the CS to P8 system.





17. Click Start Bulk Load to proceed with uploading the documents to FileNet P8.

18. Optionally, you may wish to save your XML file of data locally before starting the migration by selecting the Import/Export tab and selecting the drop down.




20 Step 3 - Migrate CS Folders (Optional) | BowenECMSolutions.com

19. As progress is made during the bulk load - migration P8 ID's will be returned for each item as it is added to FileNet. The progress bar will also indicate the progress percentage.

Step 3 - Migrate CS Folders (Optional) As an optional step, you may choose to migrate your Panagon CS Folders to P8 and file migrated documents. Filing occurs only after you have migrated documents to P8. Your folder class or subclass must have a property named CSFolderID. 1. From the main menu click File

>> CS Folder Migration





2. Enter your user name, password and P8 URI.

3. Click Logon, which typically takes 15-30 seconds to complete depending upon the activity in the P8 system and system resources.

4. Click the desired object store and folder class.

5. The folder class must have a string property named 'PanagonFolderID" else the operation will fail.





6. In the lower left list box, right click to open the context menu. Click Documents Load XML.

7. Select an XML file of data that was used during your class migration.





8. A log file is generated in the first line, take note of the path.

9. Folders are retrieved from the CS System and mapped to the documents in your XML.

10. The P8ID in the XML is used to instantly obtain the document object during the actual folder - Filing operating.




24 Step 4 - Folder via the User Interface (Optional) | BowenECMSolutions.com

11. Click the Folder View tab to view the proposed folders.

12. In this simple example, users filed documents in the SEARCHES folder and is the only folder present.

13. As folders are processed, the folder icon will change from a folder + add symbol to a plain folder icon.

14. To begin the filing operating,

click CREATE - Check CS Folders.





Step 4 - Folder via the User Interface (Optional) As an optional step, you may file documents quickly and easily into the P8 folder(s) of your choice via the CS Migrate Documents interface. You can folder documents after migration in the same session or load an XML file. The following instructions show importing XML batch file. 1. From the main menu click File >> CS

Migrate Documents





2. Enter your user name, password, P8 URI-URL and click Logon which will take 15-30 seconds depending upon the activity in your P8 system and available resources.

3. After the logon is complete, select an object store and document class.





4. Click the Data Export / Import tab, then select the import action from the drop down.

5. Click GO to execute the action and load the XML file. The grid will populate with data.





6. Click the Folder Operations tab, then right click in the clear white space area to load the P8 folders into view.





7. The P8 folders will be retrieved. Click on each folder to retrieve sub folders.

8. Highlight the desired folder you wish to file the document into. You must have filing permissions to the given folder.

9. Use the column grouping features to

aggregate documents by any values you choose. Click the custom filter to use advanced options such as starts with, ends with, contains, like, not like, greater than less than and regular expressions.

10. Highlight the rows you want to import by using the SHIFT + CTRL or CTRL keys.





11. You can easily select all rows by clicking the row selector (the cell that contains the row number) and the last row while holding the shift key.

12. In this example, the user has selected several documents manually as highlighted in the grid.

13. Click File Documents to begin the filing operation.

14. Documents will be filed and the progress bar will continue to show progress until 100% is reached. Note that due to rounding of numbers the progress bar percentage may remain at 100% while documents are filed. For instance, (current row / total rows) * 100.




31 Step 5 - Migrate Annotations (Optional) | BowenECMSolutions.com

Step 5 - Migrate Annotations (Optional) As an optional step, you may migrate annotations from CS to FileNet P8. The documents are first migrated to FileNet P8 from CS. Then launch this form to create the annotations. Annotations are migrated to P8 by locating the appropriate migrated document(s). Using the same XML batch file from a bulk load 'batch' with populated P8 ID's the program will use those unique identifiers to locate the document in P8 and create the annotation object. If the provided ID is not found the program will report this as an exception. The second method of creating annotations will search FileNet P8 for the document object by property values which are PanagonDocID and Major version number. You must pay attention to this requirement if you have allowed deletes in FileNet P8 or performed partial migrations. You can run the create annotation routine as many times as you like, the program will not duplicate annotations. 1. From the main menu click File >> CS

Annotation Migration as shown.





2. Update the P8 logon and URI path then click logon.

3. Connecting to FileNet P8 will take a minute or two, depending upon the activity in your P8 system and available resources.

4. A red blinking tool tip will prompt you to select an object store then annotation class. Select the appropriate object store where you migrated documents to and the annotation class.





5. Right click in the annotation list box and the menu will appear. Click get annotations and browse to a directory that contains the annotation objects (*.bin) files.





6. The Annotations will now be loaded into the annotation list view.





7. The same XML file that is used to migrate documents can also be used here to load annotations. After you migrate documents to P8 the XML file will contain the P8 ID of the CS Panagon document and CS version number.

8. If you have a batch XML file with P8 ID's use the import XML option. You can migrate partial documents, partial annotations and also repeat the create annotation routine as many times as necessary.

9. The program will look for a matching document by ID or (PanagonDocID+MajorVersionNumber) for each annotation that was provided. If the document is found the document is checked for an existing annotation, if it matches a new annotation will not be created.

10. Create multiple annotations per document is supported.





11. Final program output is shown. Annotations that are created will have a check box in the list view, bottom left corner.

