TUTORIAL - Importing, Editing and Deleting - · PDF fileTUTORIAL - Importing, Editing and Deleting ... Using the ESdat Environmental Database System . ... The Add New action

TUTORIAL - Importing, Editing and Deleting

For use with ESDAT version 5 (version 5 software updates available from www.esdat.com.au)

Using the ESdat Environmental Database System

Function Tutorial Importing, Editing and Deleting Data` ww.esdat.com.au

Copyright @ EarthScience Information System P a g e | 1

Function Tutorial – Importing, Editing and Deleting

Table of Contents

Summary ....................................................................................................................................2

Pre-requisites .............................................................................................................................2

Key Learning...............................................................................................................................2

Introduction ...............................................................................................................................2

The Import View ........................................................................................................................4

Import Templates.......................................................................................................................7

Import Options...........................................................................................................................9

Exercise 1: Import Templates - Adding New data................................................................11

Exercise 2: Import Templates - Updating or Overwriting data ............................................13

Editing and Deleting.................................................................................................................16

Exercise 3: Editing data ........................................................................................................16

Exercise 4: Deleting Data......................................................................................................16

Creating Drop-Downs...............................................................................................................18

Exercise 5: Creating a custom drop down menu .................................................................18

Importing documents ..............................................................................................................20

Exercise 6: Importing documents.........................................................................................20

Emailing Import Templates......................................................................................................22

Further Tutorials ......................................................................................................................22



SummaryThis tutorial:

highlights the general principles of importing data into ESdat using templates

illustrates methods for editing, overwriting and deleting data

uses the Sample Hydrogeology and Contam database (and therefore can be usedwith the demonstration version of ESdat)

is completed using the Access version of ESdat (guidance comments as to differencesbetween Access and SQL are provided within this tutorial)

requires approximately 90-120 minutes to complete

Pre-requisites

This tutorial also forms a useful pre-requisite to subsequent ESdat tutorials which describehow to import specific data-types into ESdat (such as laboratory data, environmentalstandards, and historical chemistry).

For a full list of the Tutorials available for use with ESdat, please see the Tutorial List.

Key LearningFollowing completion of this tutorial, the user will have developed skills in the following:

Using Import Templates to import data into ESdat

Editing imported data within ESdat

Deleting (removing) imported data from ESdat

Overwriting imported data within ESdat

Creation of drop down lists to encourage accurate data entry

Managing documents using ESdat

IntroductionESdat users typically have a range of data types to be managed, including:

Sample location information

Co-ordinate information (such as survey data and GPS data)

Field instrument measurements (including water levels, NAPL thickness, pH, EC etc.)

Laboratory analytical data (either laboratory reports in in ESdat format or historicaldata)

Laboratory request and Chain of Custody information

Soils descriptions and geotechnical soils data

Borehole logs and groundwater well construction details

It is assumed the user has previously completed Tutorial 2 – Introducing ESdat.

http://www.esdat.net/esdathelp/index.html?tutorials.htm



Geo-referenced basemaps in a variety of formats

To maintain integrity of a database, data must be imported into ESdat in a consistent andcoherent way.

Completing this tutorial will assist ESdat users in effectively importing data sets that may notbe in the correct format or is incomplete. Importing, editing and deleting data are key skillsfor efficient use of ESdat.

In addition, maintaining a record of source documents (such as historical reports) can beuseful if reference to original data is required for verification.



The Import ViewAn array of specifically formatted ‘Import Templates’ are available to allow importation ofvarious data types into ESdat.

These Import Templates are accessed through the Import view.

Open ESdat

Select the Sample Hydrogeology and Contam database from the project list

Select Import from the View Type tool bar (the top tool bar), as shown below

The Import view (as shown in the figure below) provides a range of Import Options forimporting data and/or information into ESdat.

The blue buttons at the left of the Import view labelled Blank Import Templates provideseight buttons to access templates for the major types of data imported into ESdat:

Locations (sample location details, such as soil bores, test pits or sediment samples)

Wells (groundwater well construction details)

Field Chemistry (or other field parameters, such as temperature)

Historical Chemistry (as distinct from laboratory supplied analytical reports)



Groundwater levels (including NAPL measurements)

Time-variant TOC (where groundwater well measurement reference points (i.e. TOCor Top Of Casing) have varied over time through damage repair, subsidence etc.)

Borehole/Geological logging - Standard (i.e. environmental) and Geotechnical logs

The scrollable list at the bottom of the blue areas provides a list of all available importtemplate data tables and reference tables (reference tables are labelled with a prefix:zRef_). This list corresponds directly to the list of tables visible in Data Views when hoveringthe mouse over Data Tables (top right).

The reference tables refer to existing information (data) that has been imported into ESdatto allow specific functionality.

For example, reference table zRef_Environmental_Standards includes third party publishedor user created assessment standards that have been imported into the database. Suchstandards (or criteria or guidelines) can be used to create analytical data summary tablescomparing analytical data against the assessment criteria.

The Files buttons within the Import view (as shown above) allows navigation to import arange of specific file types, including:

Laboratory reports (as supplied by an analytical laboratory in ESdat format)

Logger data

Populated Import Templates (i.e. templates from the blue area that have beenpopulated with data and saved to a directory)

Data Interchange file (for transferring data from one database to another)

Documents or document directory pathways stored in ESdat

AGS format files (geotechnical and geo-environmental file format)

Clicking on any of these buttons will allow navigation to the directory location of theparticular file type to be imported into ESdat, with the exception of the Documents button.



The Documents button will open a Blank Import Template for entering the details of one ormore documents.

The Library group of buttons allows users to:

Navigate to and import environmental standards saved in a directory location

(either downloaded from http://www.esdat.com.au/Environmental_Standards.aspx orthose created by the user)

Run existing or create new Post Import Calculations

Run existing or create new Combined Compounds

(Users should refer to the ESdat Users-Manual for details on these two advanced concepts)



Import TemplatesClicking on any of the buttons in the blue area of the Import View will open a Blank ImportTemplate (i.e. a formatted excel spread sheet).

Import templates will vary in appearance, depending on the data type that has beenselected.

Open the Locations Blank Import Template (by clicking the Location button, asshown below, or from the scrollable list)

Note the first few rows of the Locations Import template is shown in the figure below:

The column headings are referred to as Field Names.

Return to the Locations template and note two cells are coloured green

In the Locations Import template, cells A3 and A4 are both coloured green.

Primary Keys

The values in a column headed by two green cells are known as Primary Keys.

In the Locations Import template, green cells A3 and A4 define the Primary Key Field NameLocCode. (SQL users note - the Primary Key Field Name is Location_Code).

All values in this column (i.e. from cell A5 and below) are Primary Key values.

Primary Keys values are always unique.

LocCode (Location_Code in SQL) is an abbreviation for Location Code.

An example of LocCode might a borehole ID e.g. BH01.

Only one record can exist for each LocCode per ESdat Site/Project.

Data specific to BH01 is entered in the corresponding row within the Locations ImportTemplate, filling the relevant cells to the immediate right of the LocCode, BH01.

It is not possible to import two records with the same LocCode (for example, there can onlybe one BH01).

A second (unique) LocCode may be BH02, a third may be BH03, and so on.

This uniqueness is important, as it ensures the correct data (for example analytical data)relates to a unique sample location.



Open the Wells Construction and Field Parameters Blank Import Templates byselecting the relevant buttons within the blue area

Review the Field Names within the Wells Construction and Field Parameters importtemplates

Note the different Field Names for different data types.

Close the Wells and Field Blank Import Templates

Combination Primary Keys

Open the Water/NAPL Levels Blank Import Template

In the Water/NAPL Levels example (as shown in the figure below), several fields combine toproduce a Primary Key for each record (i.e. each row).

In this case, there can be only one record for a specific combination of:

Location (LocCode column)

Well (WellCode column)

Date/Time, &

Measurement Method

This will form the Primary Key for the record (with the record being, for example, a gaugedwater level of one well on a single occasion, which is a unique measurement).

Measurement method is included as it is possible that a dip measurement and loggermeasurement may occur at the same time in the same well.

A single green cell means the data is required in that column for each record (i.e. each row),although it may be populated by a default value.

For the Dry field there are only two options available (true or false), with the default being isthe well is not dry (false).

Where a field is populated by a default, a comment is included with the field name (hoverover the red triangle). The Measurement Method has a default.

In order to further explore the use of templates to import data, the example data providedwith the ESdat installation will be used.



Import OptionsThe ESdat import process is described in two short exercises that are presented later withinthis tutorial.

Prior to commencing with the examples, it is important to understand the process of howthe data can be imported into the database.

Upon populating a blank Import Template with the required data, it requires importing intoESdat by clicking the Import button from within the Import template.

A dialogue box entitled Upload Data will appear (shown below), presenting various ImportOptions controlled by two drop down lists.

The Record Action drop down list manages the way the imported data is entered into ESdat.

The Add New action (as shown in the figure below) will only import data for a new PrimaryKey (e.g. a single water level measurement taken at a specific time and date cannot beimported twice).

Overwrite Existing instructs ESdat to find the existing record for that Primary Key (e.g.assuming the water level measurement that has already been imported) and overwrite allvalues for that record (i.e. the relevant row).

If a field in the data being imported is left blank it will overwrite any existing values in thedatabase with a blank. Overwrite Existing should generally be used if there is no confidencein the accuracy of any of the data within the specific record.

Update Existing operates similarly to Overwrite except that any blank cells in the ImportTemplate are ignored, thus any existing values in the database are not overwritten with ablank value in the import template. This is generally safer than Overwrite.

Use Update Existing to correct errors in single fields or to populate blank fields within thedatabase (e.g. adding comments).

Add New and Update Existing will add new records to ESdat by importing data for a newcombination of primary key values AND search for an existing occurrence of the primarykey values (if records are found with the same Primary Key), and then update the record



with any new data found from the import template. (Please note - this option is onlyavailable for specific Import Templates, such as Environmental Standards)

Require Parent Records means that higher level (Parent) data must be present.

This option is useful if a complete list of parent records, such as Location Codes and/or WellCodes exist in the database and consistent naming is required when entering associateddata.

An example would be avoiding entering water levels for GW1 (the incorrect parent record)rather than GW01 (the correct parent record).

Create Missing Parent Records will create higher level data if it is missing.

For example, if water levels are being entered for a Well that has not been previouslyimported into ESdat, a corresponding new Location Code and Well Code will be created inthe database if Create Missing Parent Records is selected.

For more information on importing specific Data-Types see the tutorials on specific topicssection.



Exercise 1: Import Templates - Adding New dataIn this exercise, water level data will be imported to the Sample Hydrogeology and Contamdatabase for a well that already exists within the database (BH08A).

Open ESdat

Select the Sample Hydrogeology and Chem database (see here for assistance ifrequired)

Click the Import View Type button (from the top tool bar)

Click on the Water/NAPL Levels button (within the blue Blank Import Templatesarea)

The following Blank Import Template will open in Excel:

The data shown below will now be transcribed into the template, then imported into ESdat:

LocCode WellCode Date_Time Water_Elevation Units CommentBH08 A 8-May-10 26.98 m Test Comment 1BH08 A 8-Jun-10 26.98 m Test Comment 2

Copy and Paste the data shown above (or type manually) into the relevant columnsin the GW/NAPL Data Blank Import Template

Arrange the data in the correct columns

Go to the Add Ins tab at the top of the page and click on this to open the CustomToolbar (as shown in the figure below)

Click on the Import button located within the Add Ins tab

Select Add New from the Record Actions drop down and Require Parent Recordsfrom the Parent Tables drop down (as BH08A already exists in the database)

http://www.esdat.net/downloads/Tutorials/Functional_Tutorial_Opening_the_ESdat_Sample_Hydrogeology_and_Contam_sample_dataset.pdf



Click Import

Data will now populate the database. Note the Import Information and Import Errors boxes.

The template should appear consistent with the figure below. The light green shadingindicates rows that have been successfully imported. Purple indicates an import error.

Hovering over the purple cell with a comment will reveal the cause of the import error, asshown below.

One data row has been imported to the database with no errors (shaded pale green), butthe other (shaded purple) will cause an error.

The comment in Cell A5 (indicated by the red triangle) indicates that the record alreadyexists for this specific water level Primary Key (i.e. LocCode + Well Code + Data/Time).

In the case of such errors, it is useful to compare the data to be imported with the dataalready in the database.

To compare the data to be imported with the data already in the database complete thefollowing:

In the Upload Data dialogue box select the option checkbox Show comparison if thedata being imported already exists (as shown in the figure below)

Click Import



(the dialogue box should still be open, if not click again)

A new (separate) worksheet will be created (note: at the bottom of the Excel window lookfor the tab labelled WaterLevel_Comparison).

Select the WaterLevel_Comparison tab

The data to be imported will be shown to the left (under the title Import Data). The ExistingData in the database is shown to the right (scroll to the right to see all – refer to the top rowof the sheet highlighted yellow for guidance).

Differences between the data already the database and the new data to be imported will behighlighted in purple (note: some differences are fields which have been filled in with adefault).

Select the WaterLevel tab in Excel (bottom of Excel window) and leave Excel open forthe next exercise

Exercise 2: Import Templates - Updating or Overwriting data Return to ESdat

Select to Data Views from the top View Type tool bar (as shown below)



Select WL1_WaterLevels_AHD from the Water Levels drop down list (hover over theblue Water Levels button) to view the Water Level information in ESdat

Scroll down the data and double click on BH08 in the LocCode column

Double click on A in the WellCode column

Click Apply Filter

The data should appear as it does in the figure below.

Note that only the row that was imported from the Excel Import Template has the commentin the data.

Return to WaterLevel tab in Excel

If required, Click on the icon in Excel to reopen the Upload Data dialogue box

Change Record Action to Update Existing

Click Import

The row that was previously purple in the spreadsheet is now shaded light green, indicatingsuccessful import of data.

Return to ESdat

click Apply Filter (to refresh the data)

Confirm that both comments are now listed in the data in ESdat.

Return to Excel

Click Import

As the data is shaded Green, ESdat considers the data is already successfully imported andcan be ignored.



Remove the Green shading using normal Excel tools

Change Record Action to Add New

Click Import

A Record already Exists error dialogue box will appear, as this data is alreadyimported

Delete the comments Test Comment 1 and Test Comment 2 from the data in theImport Template

Import with Update Existing selected

Return to ESdat

Click Apply Filter (to refresh the data)

Confirm that both comments are still listed in the data in ESdat (recall that Update Existingwill not replace existing data with blanks from Import Templates).

Remove the Green shading using normal Excel tools and retry the import, this timeusing Overwrite Existing

Switch back to ESdat and click Apply Filter (to refresh the data).

Confirm that both comments are now missing in the data in ESdat. Overwrite Existing willreplace existing data with blanks from Import Templates.

Close Excel without saving

The Add New, Overwrite Existing and Update Existing Record actions in the Import Optionshave now been successfully applied.



Editing and DeletingData within ESdat can be edited from within the database or removed completely.

The following two exercises will demonstrate both editing and deleting, using the same dataas used in Exercises 1 and 2 and applying the same filter.

Exercise 3: Editing data Return to ESdat WL1_WaterLevels_AHD within Data Views

Click on the Allow Editing button (the button should turn red to indicate EditMode is operating)

Select two cells in adjacent rows from the Comments column and type in a comment(e.g. test)

Press Enter (confirm yes when prompted to copy data down)

The comments will now apply to both rows.

An entire column can also be selected, and all visible cells in that column (i.e. that is shownafter filtering) will have the new value.

Select the whole Comments column (click the grey header cell at the top of thecolumn, which contains the column name)

Type some text and press Enter - the whole column will update

Click Remove Filter

Scroll down to notice only the rows that were previously filtered (i.e. LocCode BH08 andWellCode A) will have comments added.

Exercise 4: Deleting DataIf required, entire rows of data can be deleted from ESdat.

Using the same data as used in Exercise 3 (recreate and apply the LocCode BH08 andWellCode A filter if required – Edit Mode will need to be switched off):

Click on the Allow Editing button (if not currently in Edit Mode)

Left click on the grey row-header on the far left of any row with a LocCode value ofBH08 to select that row (will highlight blue if selected correctly)

Right click on the identical far left position

Select Delete Row (confirm editing when prompted)

The row will be deleted. Note other data related to BH08 still remains (this is important forthe following steps).

Select the Locations Data-Type button (from the row of blue buttons above thefilter)

Click on the Allow Editing icon



Left click on the grey row-header on the far left of the row with a LocCode of BH08 toselect that specific row

Right click on the identical position

Select Delete Row

An error message will appear indicating the record cannot be deleted as another table hasrecords relating to this specific Location.

This means that other data within the database (such as water level data) relates toLocation BH08 (the Parent Record), and hence it cannot be deleted from the Location table.

This other related data for BH08 must be explicitly deleted prior to being able to delete theParent Record, Location BH08 from the Locations Data View.



Creating Drop-DownsIt may be useful to create drop-down data entry menus, in order to encourage consistentdata entry (such as standard naming conventions).

For example, particular purposes for soil bores may be required, such as either targeted orgrid based.

A second example may be offering a choice between aquifer units being monitored bygroundwater wells.

Drop-downs can be populated is through use of the zRef_Drop_Downs table.

Other approaches are also available (see the ESdat Help files).

Exercise 5: Creating a custom drop down menuIn this scenario, monitoring is being conducted for two purposes:

1. LIC_Test (for licence testing) and

2. COUNCIL_Test (for local council testing).

This exercise uses the same Locations data from the Sample Hydrogeology and Chemdatabase as used in the previous exercises.

Click the Locations Data-Views button

Right click on any cell in the Purpose column

Select Add to zRef_Drop_Downs (final option)

A dialogue box labelled Create a drop-Down Value will open (as shown in the figurebelow)

A drop-down value for Location.Purpose is required, which must be limited to less than 30characters.

Type LIC_Test for the value and click OK

Click on the same, or another cell in the Purpose column

Type COUN_Test in the drop down value dialogue box and click OK

Click on the Allow Editing icon (as per Exercises 3 and 4)

Click on any cell in the Purpose column and a drop down menu arrow will appear (asshown in the figure below)



Click the drop-down, and the two entered options should appear

The change made above will carry over to the Locations Import Template.

To see this:

Click on the Import view

Select the Locations Blank Import Template

The Import template will be generated in Excel, this time offering the two options forPurpose just created (as shown in the figure below)

Close the template without saving

It should be noted that while users can select from the options presented in the drop downlist, any entry can be typed into the cell.

As stated at the start of this section, drop down menus can be used to encourage consistentdata entry, however they cannot be used to enforce consistency.



Importing documentsDocuments can be stored and retrieved through ESdat databases. Electronic documents ofmost types, such as MS Office or PDF documents (such as historic reports) and photographs(such as aerial photographs or site photographs).

There are differences between the Access and SQL versions of ESdat for storing documents.

In Access versions, only the path to the file is stored in the database. The directory locationof these documents is determined at the users’ discretion.

The SQL version of ESdat has two options for storing documents:

1. Document_Path can be used to simply store the path to the document in ESdat2. Document can be used to store a copy of the document in ESdat

The SQL Server version of ESdat can be configured (using Setup - Document Management)to store documents either internally in the database, or in a server folder.

Exercise 6: Importing documentsThe following exercise uses the Access version of ESdat.

Within ESdat, navigate to the Import view

Click on the Documents button in the orange Files area

A template will appear with the following columns (as shown in the figure below):

o LocCode: will offer a drop down menu for data input

It should be noted that a LocCode is not required if the document doesn’t relate to a specificlocation.

o Document Type: e.g. ‘site inspection photo’, ‘report’, ‘scanned field notes’etc.

o Description: e.g. ‘1965 aerial photograph’ etc.

o Document_Path: Double click the cell to open a window to navigate to therequired file, or type in the document file path directly

Click the Import button (under Add Ins)

Records of documents stored in ESdat are viewed by:

Navigate to Data Views



Select the Documents output listed under the Locations Data-Type button (as shownbelow)

Double clicking on the Document Path (or Document in SQL Server) field will launch thedocument.



Emailing Import TemplatesBlank Import templates, generated as Excel files, can be emailed to other parties (field siteoffices, consultants’ offices etc.) to allow population with data.

To do this, create a blank Import Template in Excel (via the Import View), and attach it to anemail.

Alternatively, a ‘Send Template to email’ checkbox box is at the bottom left of the bluesection of the ‘Import’ page.

If this is checked, a newly created Template it will be automatically attached to a new emailthat will open and be ready to send.

(Note: applies only to MS Outlook email, configured to support external applicationsattaching files.)

Further TutorialsFor a list of other tutorials available with ESdat seehttp://www.esdat.net/esdathelp/index.html?tutorials.htm