WAS to Archive-It Metadata Migration

March 11, 2015

WAS -> Archive-It

WASProject/Archive

• 3 levels of hierarchy– Project– Site (can contain 1 or more Seed URLs)

– Seed URL

Archive-It Collection

• 2 levels of hierarchy– Collection– Seed URL

2 Seed URLs per Site

1 Seed URL per Site

1 Seed URL per Site

1 Seed URL per Site

2 Seed URLs per Site

Multiple seeds – flattens out; each Seed URL gets all the Site Metadata

BEFORE starting, you should…

Delete sites (seeds) that you have never captured or you captured, but you deleted all the captures. Probably sitting under ‘never captured’ or ‘inactive sites’

How to move

• Move project (collection) by project (collection).

• When you sit down, start and finish the move of a project.

• You don’t have to do all projects/collections in one day

Run two reports (Administration > Project Admin)1. Click “Archive-It Seed Export” > Export Seeds2. Click “Archive-It Seed Metadata Export > export metadata

Coming Soon in your accounts

Export Seeds

Seeds export from WAS

• It is in .txt format, open it with notepad • Your seeds will be segmented by crawl

frequency. • E.g., “Seeds with custom schedule of 1x per

year”• You will copy and paste URLS from the .txt

document and upload them in chunks by frequency

Example text file

Consult the WAS- Archive-it mapping document to decide on the equivalent frequency

https://wiki.library.ucsf.edu/download/attachments/351243364/MappingofWAStoArchive-It.pdf?version=1&modificationDate=1422304077000&api=v2

Create a Collection (project) in Archive-It

Create Collection (aka Project)

Select frequency: for now leave at “One-Time’, click next

Enter Collection level metadata. This metadata displays in the public site. You can go back and

fully enter this later

Topics will appear in public site(along with any Subjects you have)

Example display

In order to create a collection, you must upload seeds.

If you have Historical seeds, Upload those FIRST(!)

• Historical sites/seeds are seeds where the seed URL has changed over the life of the captures.

• They will be at the top of your seeds .txt document• Do these first because it is easiest to do a ‘bulk edit”

and select ‘deactivate”

Example seeds list with Historical Seeds

Copy and paste seeds from .txt fie into box. Leave ‘Default’ selected > Next

VERY important:1. Ignore this error for ALL your seed uploads. 2. “URL is correct; use as is” MUST be checked regardless of the error you see. If it is not selected for any seeds, go thru now and change it for all instances.

Another example, click: “URL is correct; use as if” for all

Collection created

Bulk Edit Historical Seeds (where applicable)

Under “Seed Management” click “All”

Click top box to select all. Note: you will ‘select all’ for what is displayed, if there are more than 400 items,

they are on another page. You will have to repeat

Click “bulk edit”

Choose “Deactivate”

Go back to bulk edit > Add Metadata

• Suggestion: add a Notes field if you don’t already have one, where you note that these are historical seeds. Most likely will never want to crawl these again so you may want to keep track

Add a custom field

Go back Collection management and repeat for the next frequency in your seed list

Back to Seeds .txt file

Leave as ‘one-time” they will not crawl until you say crawl now

Copy and paste seeds into box. Leave ‘Default’ selected > Next

For this case, choose Quarterly

Import metadata

Click “ALL seeds” > Import metadata

Upload the metadata file > Upload File (leave default setting)

You could stop here and do the clean up at a later day

Metadata cleanup

• If there is a WAS field that is not in Archive-it, on import Archive-it creates a custom field.

• All fields will display in the public interface by default

• The following fields may be in your upload, but they should ALL be made private:Note, Scope, Robots honored, Max crawl seconds, Capture frequency, Seed type, Site ID

How to make fields private in Archive-it:

1. Go to Admin (link in the upper right corner)2. Account Settings 3. In the text box toward the bottom of the page

called 'Private Metadata Fields' enter all these fields: Note , Scope, Robots honored, Max crawl seconds, Capture frequency, Seed type, Site ID

4. NB: Enter each field name on a separate line, in all lower case letters.

Scope –> Seed Type

• What about Directory only?• What about Page only?

NB. Archive-it offers a lot of additional scoping options for crawls. View: Help Documentation (linked top, right of collection page)

Directory is not a separate scoping option in Archive-it ( it is handled through slash - /)

NO action need by you, except to QA

WAS – Directory crawls

• Rosalie.com/presentations– We will add the ending slash for you if you didn’t

• Rosalie.com/presentations/– It moves over as is

• Rosalie.com/presentations.html– It will crawl as host

What about ‘page only’ crawls?

• For ‘Page only’ you will have to manually go back and change crawl scope (seed type)

• You can find these by opening the metadata export. It is in .ods format, which you can open in Google docs, with most versions of excel or download open office.

• Do NOT edit the .ods file before doing the metadata upload; make a copy.

• Then sort “scope” column to find the relevant URLsHow to change it:• Page: click on Settings > Crawl one page only (can also be

bulk edited)

Change Frequency under Settings > Seed Type

When will my crawls start? When you start them.

When do I shut off WAS crawls?

• FIRST set up your crawls in Archive-It• Make sure daily crawls are running• Then you can stop your WAS crawls

VERY important: Do NOT make any edits to WAS data, crawls, ANYTHING once you have moved a project to Archive-It!

Batch shut off crawling in WAS

Sites > Manage Sites > “all” > “select all” > “Reschedule Selected”

Select “off” and click “Reschedule”

Send CDL your info

After you have created all your collections,

1. Send Rosalie this info for each collectiona) Collectionidb) Accountid

AND

2. Add Rosalie as a user to your account (for now)

CollectionId and AccountId in URL

Where’s my data?

• Archive-It will work with CDL staff to move over your data.

• Timeline: May/June 2015

Resources

WAS – Archive-It Migration wiki: https://wiki.library.ucsf.edu/display/UCLCKG/WAS+-%3E+Archive-it+Migration

Mapping of terms and metadata: WAS - Archive-It:https://wiki.library.ucsf.edu/download/attachments/351243364/MappingofWAStoArchive-It.pdf?version=1&modificationDate=1422304077000&api=v2