View
214
Download
0
Category
Preview:
DESCRIPTION
Q5 Highlights (II) Testing: More tests are always added with new features. Tests verify correctness of asynchronous execution, axe dependencies, function shipping, and HDF5 to IOD translation (to a simple extent). Try to cover all routines added. Automated with a simple script to run tests on a local machine. Still working more on a larger automated regression testing framework. (should be available in Q6). This Demo will focus on 2 main and new features added: Transactions and Read Contexts Dynamic Data Structure support (Map objects) Copyright © 2013 The HDF Group. All Rights Reserved 3
Citation preview
www.hdfgroup.org
The HDF Group
1
The HDF Group Q5 Demo
5.6 HDF5 Transaction API5.7 Full HDF5 Dynamic Data Structure
Copyright © 2013 The HDF Group. All Rights Reserved
www.hdfgroup.org2
Q5 Highlights (I)
• New features added since Q4 (other than 5.6 & 5.7):• Support for variable length datatypes• Some deferred items from Q4 (mostly metadata get routines)• Data Integrity for metadata; although not helpful at the
moment without a real storage backend.• Enable/Disable Data integrity checks through a property.
• Can be done for metadata and raw data.
• Still working with skeletal IOD on a local Linux box:• EMC has their first code demo this quarter.• Many tests (especially reads) are “faked” for now.
• This is the main limitation for adding automated regression testing framework and benchmarks.
• Accounts for new Cray system at LANL pending.
Copyright © 2013 The HDF Group. All Rights Reserved
www.hdfgroup.org3
Q5 Highlights (II)
• Testing:• More tests are always added with new features.
• Tests verify correctness of asynchronous execution, axe dependencies, function shipping, and HDF5 to IOD translation (to a simple extent).
• Try to cover all routines added.• Automated with a simple script to run tests on a local machine.• Still working more on a larger automated regression testing
framework.• (should be available in Q6).
• This Demo will focus on 2 main and new features added:• Transactions and Read Contexts• Dynamic Data Structure support (Map objects)
Copyright © 2013 The HDF Group. All Rights Reserved
www.hdfgroup.org4
Outline
• 5.6 HDF5 Transaction API:• Transactions and Versions recap from earlier technical
presentation.• Diving more into semantics and usage.• Go through some pseudo code & actual code.• Run Demo.
• 5.7 Full HDF5 Dynamic Data Structure:• Changes from Q4• New Map objects and routines• Code example and Run Demo.
Copyright © 2013 The HDF Group. All Rights Reserved
www.hdfgroup.org5
FastForward Transactions (I)
• A transaction consists of a set of updates to a container.• container ≈ file• Updates are added to a transaction, not made directly to a
container.• Updates include additions, deletions, and modifications.
• Any number and size of updates may be included in a single transaction.• Tiny transactions may have high overhead.• Large transactions will amortize the overhead.
• Multiple processes can add updates to a single transaction.
Copyright © 2013 The HDF Group. All Rights Reserved
www.hdfgroup.org6
FastForward Transactions (II)
• A transaction is finished when no more updates will be added to the transaction.• Transactions can finish in any order.• The updates for a finished transaction are not visible in the
container.• A finished transaction must be committed in order for
its updates to become visible in the container.• Transactions are committed in strict numerical order.• When a transaction is committed, all updates in the
transaction are applied atomically to the container and become visible.
• If all updates cannot be applied, none are applied, and the transaction is discarded.
Copyright © 2013 The HDF Group. All Rights Reserved
www.hdfgroup.org7
FastForward Container Versions
• A version identifies a container at a given state.• The version number equals the number of the
committed transaction that transitioned the container into the state.
• A read context can be created for a container version.• The read context allows access to the contents of the
container version.• A given container version is guaranteed to remain
readable until all associated read contexts are closed.
Copyright © 2013 The HDF Group. All Rights Reserved
www.hdfgroup.org8
Acquiring a Container Version
• This is important, so that IOD will not flatten this version, and it is available to read from.
• Only one process is required to acquire, but any number can. IOD will ref. count the acquired context.• If one process acquires a version, it can communicate the
acquired version to other processes so that they do need to acquire it themselves.
• The same number of release calls need to be issued as acquire calls.• [although different processes can release than acquired]
Copyright © 2013 The HDF Group. All Rights Reserved
www.hdfgroup.org9
Transactions and Read Contexts in Q5
• Two new set of routines added to the HDF5 FastForward API:• H5TR for transactions• H5RC for Read Contexts
• All new APIs are asynchronous.• All HDF5 read/get routines take Read Context IDs. • All HDF5 write/update routines take Transaction IDs.
• Note that transactions specify a read context, so writes/updates happen within one.
• H5Fcreate_ff() will always use up transaction 0, i.e. application starts using transaction 1.
• H5Fopen_ff() will return an acquired read context with the latest readable version of the container.
Copyright © 2013 The HDF Group. All Rights Reserved
www.hdfgroup.org10
Container Version Acquire/Release semantics
Copyright © 2013 The HDF Group. All Rights Reserved
/* Any Leader Process */ version = 15;/* acquire container version 15 */ rid = H5RCacquire(file_id, &version, H5P_DEFAULT, event_q);
/* wait for the acquire to complete. This is not necessary, but user must live with the consequences that the acquire on a flattened version will fail and so all subsequent reads and the release call on rid will fail . */
/* If Leader has delegates that it wants to tell that it has acquired the container version, it has to wait for the acquire to complete before informing them */H5EQwait(event_q, &num_requests, &status);
MPI_Ibcast ()/ MPI_Isend() …
/* Read from Container */…
/* If other processes were informed to use this version, must wait to hear from them before releasing */MPI_Barrier()/MPI_Bcast()/MPI_Recv() …
/* release container version 15. This is async. */ H5RCrelease(rid, event_q);
/* Close RC ID. This is a local operation that just frees resources. */ H5RCclose(rid);…/* Wait on all Events, Everything was asynchronous thus far */H5EQwait(event_q, &num_requests, &status);
/* Any Delegate process */
MPI_Ibcast()/MPI_Irecv() …MPI_Wait();
/* client received a version – x = 15 ; create a read context ID. This is a local immediate operation. */rid = H5RCcreate(file_id, x);
/* Read from Container */…
/* Wait for all reads to complete */H5EQwait(event_q, &num_requests, &status);
/* Tell leader I am done with my reads.*/MPI_Barrier()/MPI_Bcast()/MPI_Secv() …
/* Close RC ID. This is a local operation that just frees resources. */ H5RCclose(rid1);…
www.hdfgroup.org11
Hints to H5RCacquire
• It is possible that the user is not interested with one Exact version to acquire; i.e. does not want the acquire to fail if the version specified is not valid.
• Through the Read Context Acquire property list, the user can specify the following hints to acquire:• H5RC_EXACT: Fail if the current version is not available
(default).• H5RC_PREV: Acquire the highest version smaller than the
one specified if it is not available.• H5RC_NEXT: Acquire the lowest version greater than the one
specified if it is not available.• H5RC_LAST: Acquire the last readable version; this will ignore
the version specified in acquire.Copyright © 2013 The HDF Group. All Rights Reserved
www.hdfgroup.org12
Other ways to Acquire a Version
• When opening a container, the user can optionally ask to also acquire the last readable version of the container:
• When finishing a transaction, the user can also optionally ask to acquire it into a read context:
Copyright © 2013 The HDF Group. All Rights Reserved
file_id = H5Fopen_ff(file_name, H5F_ACC_RDONLY, fapl_id, &rid, H5_EVENT_QUEUE_NULL);
H5TRfinish(tid, H5P_DEFAULT, &rid, H5_EVENT_QUEUE_NULL);
www.hdfgroup.org13
Creating a Transaction
• All operations that write/update the container must be part of a started transaction.
• All transactions must be created within a read context, because some updates require reading from the container.
• Transactions may be started by x leader processes and communicated to other delegate processes, or started by all processes.• Those are two different models of operation.• In the former case, it is not required that the processes who
started the transaction be the one who finishes or aborts it.
Copyright © 2013 The HDF Group. All Rights Reserved
www.hdfgroup.org14
Operating Models
Copyright © 2013 The HDF Group. All Rights Reserved
Leaders (Red) • Will start transaction.• Will communicate that
transaction has started and is available to do updates on to the delegates (blue).
• Will hear back from delegates that they are done updating.
• Will finish transaction; or designate a delegate to do it.
• Same number of start and finish calls.
Delegates (Blue)• Will hear from leaders that
transactions can be used.• Can write to transactions.• Have to inform leaders
when done with updates to transactions.
www.hdfgroup.org15
Leaders/Delegates Model
Copyright © 2013 The HDF Group. All Rights Reserved
/* create transaction object with an already acquired read context 15 – this is a local immediate operation */tid = H5TRcreate(file_id, rid, (uint64_t)20);
/* start transaction 20 with default model, i.e. Leader Model. */if( I am a leader process) /* This is asynchronous, but here we make it synchronous so we can tell the delegates that it has been started */ ret = H5TRstart(tid, H5P_DEFAULT, H5_EVENT_QUEUE_NULL); trans_num = 20;
/* Tell other procs that transaction 20 is started */MPI_Ibcast(&trans_num, …);
/* Leader processes can continue writing to transaction 20, while others wait for the ibcast to complete */if(I am a delegate process) MPI_Wait(&mpi_req, MPI_STATUS_IGNORE);/* Write to container */…./* Delegate processes have to complete operations before notifying the leader */if(I am a delegate process) H5EQwait(event_q, &num_requests, &status);
/* Each Leader synchronizes with its delegates that they are done writing */MPI_Barrier()/Bcast()/…if( I am a leader process) /* Finish the started transaction. */ ret = H5TRfinish(tid, H5P_DEFAULT, NULL, event_q);/* Note that leader does not have to wait for its updates to complete before issuing the finish. We track this internally. *//* Wait on all Events*/H5EQwait(event_q, &num_requests, &status);
www.hdfgroup.org16
Multiple Leaders/No Delegates Model
Copyright © 2013 The HDF Group. All Rights Reserved
/* All processes are considered equal participants in the transaction semantics; no inter-process communication is required More communication with the IONs will be done though. */
/* create & start transaction 20 with num_peers = n */tid2 = H5TRcreate(file_id, rid2, (uint64_t)20);trspl_id = H5Pcreate (H5P_TR_START);H5Pset_trspl_num_peers(trspl_id, n);H5TRstart(tid2, trspl_id, event_q);H5Pclose(trspl_id);
/* Update/write to the container */…
/* finish transaction 20. */H5TRfinish(tid2, H5P_DEFAULT, NULL, event_q);
/* Wait on all Events; everything was asynchronous thus far. */H5EQwait(event_q, &num_requests, &status);
www.hdfgroup.org17
Operating Within Transactions
• HDF5 has metadata and raw data operations.• Operations that occur inside a transaction are typically
to update the contents of the container, i.e. create a group, create a dataset within a group, write to the dataset, etc…
• Is it required, for example, if I create a dataset to commit the transaction that the dataset was created in to be able to write to the dataset?• No! But there are limitations/rules that must be followed
Copyright © 2013 The HDF Group. All Rights Reserved
www.hdfgroup.org18
Operating Inside a Transaction
• If I start transaction 1:• Create a group G1:
• Create a group G2 in G1:
• This is possible because a read is not required to write to G1. • If we happen to do something like this :
the operation will fail because it will require a read to get to G1 from the root group. But since G1 is not readable in the transaction it has been created in, the result will be a failure.
• Using a path in an access operation requires all objects in the path to be readable from the read context that is used by the transaction.
Copyright © 2013 The HDF Group. All Rights Reserved
gid1 = H5Gcreate_ff(file_id, "G1", H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT, tid1, event_q);
gid2 = H5Gcreate_ff(gid1, "G2", H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT, tid1, event_q);
gid2 = H5Gcreate_ff(file_id, “G1/G2", H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT, tid1, event_q);
www.hdfgroup.org19
Access to Objects created by Leader
• But what if 1 process created the object, and other processes need to write to the object in the same transaction?• The leader process will need to retrieve a token representing
the object when it created it. This is a local operation:• H5Oget_token(object_id, &token);
• Send the token to the other processes.• The other processes will open the object using the token
they received:• obj_id = H5Oopen_by_token(token);
• This is commonly referred to as local-to-global/global-to-local operation.
• This is not currently supported. Deferred to Q6.Copyright © 2013 The HDF Group. All Rights Reserved
www.hdfgroup.org20
Quick Summary
• Transactions provide a mechanism for making atomic updates to a container.
• Committed transactions result in container versions.• Writes are done to the future.
• To uncommitted transactions• Reads are made from the past.
• From container versions
Copyright © 2013 The HDF Group. All Rights Reserved
www.hdfgroup.org21
Demo
• Look at some example code.• Run Tests.
Copyright © 2013 The HDF Group. All Rights Reserved
www.hdfgroup.org22
Dynamic Data Structures
• The main purpose is to support ACG’s need to access the FastForward stack.
• We added a new HDF5 object called a Map object with a new set of routines that should fully support the Dynamic Data Structure use case.
• H5DO routines to conveniently append to datasets and fast-append mechanism:• Realized that implementation with IOD will not be possible without an
atomic append feature. • Will drop from current project as it wasn’t high priority as other
features.• Support for variable length data is done with some limitations:
• No nested VL types, or VL types as fields of compound types.• Neither are very commonly seen.
Copyright © 2013 The HDF Group. All Rights Reserved
www.hdfgroup.org23
Map objects
• A Map object is a direct mapping to a KV store.• We wanted to expose that type of access to the application.
• New routines added:
Copyright © 2013 The HDF Group. All Rights Reserved
hid_t H5Mcreate_ff(hid_t loc_id, const char *name, hid_t keytype, hid_t valtype, hid_t lcpl_id, hid_t mcpl_id, hid_t mapl_id, hid_t trans_id, hid_t eq_id);hid_t H5Mopen_ff(hid_t loc_id, const char *name, hid_t mapl_id, hid_t rcxt_id, hid_t eq_id);herr_t H5Mset_ff(hid_t map_id, hid_t key_mem_type_id, const void *key, hid_t val_mem_type_id, const void *value, hid_t dxpl_id, hid_t trans_id, hid_t eq_id);herr_t H5Mget_ff(hid_t map_id, hid_t key_mem_type_id, const void *key, hid_t val_mem_type_id, void *value, hid_t dxpl_id, hid_t rcxt_id, hid_t eq_id);herr_t H5Mget_types_ff(hid_t map_id, hid_t *key_type_id, hid_t *val_type_id, hid_t rcxt_id, hid_t eq_id);herr_t H5Mget_count_ff(hid_t map_id, hsize_t *count, hid_t rcxt_id, hid_t eq_id);herr_t H5Mexists_ff(hid_t map_id, hid_t key_mem_type_id, const void *key, hbool_t *exists, hid_t rcxt_id, hid_t eq_id);herr_t H5Miterate_ff(hid_t map_id, hid_t key_mem_type_id, hid_t value_mem_type_id, H5M_iterate_func_t callback_func, void *context, hid_t rcxt_id);herr_t H5Mdelete_ff(hid_t map_id, hid_t key_mem_type_id, const void *key, hid_t trans_id, hid_t eq_id);herr_t H5Mclose_ff(hid_t map_id, hid_t eq_id);
Recommended