PowerPoint Presentation Template 2 Services we provide to BioHPC user
– BioHPC cluster
– Basic File Management
Slides and guide online at:
https://portal.biohpc.swmed.edu
Image retrieved from http://giannisgiataganas.blogspot.com/2010/06/lets-talk-about-cloud.html, http://www.nec.com/en/global/solutions/cloud/portfolio/storage.html
Cloud storage (one of the fundamental components of cloud computing) is a service model in which data is maintained, managed and backed up remotely and made available to users over a network (typically the Internet).
Cloud computing refers to the delivery of computing resources over the network.
Security
4/32
Why We Need the BioHPC Cloud Service
UTSouthwestern General Cloud Policy: UTSW information must never be stored in unapproved online file storage systems.
OneDrive for Business – 1TB Free for UTSW users, but max file size is 10GB, slow. vs
BioHPC Cloud: 150GB + access to BioHPC storage systems >2.5PB
• Max upload/download speed ~ 10 MB/s • Web browsers do not support folder uploading • Upload/download limit for single file (size)
50 GB
• BioHPC File Exchange (web-interface) https://cloud.biohpc.swmed.edu 50 GB / user, local storage
• BioHPC Lamella Cloud Storage (web-interface), on campus only, private cloud https://lamella.biohpc.swmed.edu 100 GB / user, local storage Gateway to BioHPC Cluster (via FTP, SAMBA or WebDAV*)
* Can be increased on PI request with Dept. Chair approval.
6/32
Some core facilities provide access to BioHPC for their users to transfer data etc.
The core decides the amount and type of storage to provide to their users, e.g.
TIBIR WBMF Core:
/project/TIBIR/WBMF_Core/<username> 250GB / core user
This is also your home directory No separate home2 or work space
Storage allocation and usage is at the discretion of the core, not BioHPC.
BioHPC Storage Backup
Since January we perform mirror back up on:
/home2 Twice weekly (Mon & Wed) 2 copies – home2 usage counts 3x against lab storage allocation
/work Weekly (Fri->Sat) 1 copy – work usage counts 2x against lab storage allocation Excludes some large users
Mirror backups Copy of your files at the time the backup runs. No old versions.
Email biohpc-help@utsouthwestern.edu to recover files.
Only requested files – not all of /project !
PI needs to request backup to biohpc-help@utsouthwestern.edu Let us know which directories to backup, and how often to run the backup.
Weekly incremental backup by default. Old versions of files are kept available.
Backup usage counts against lab storage allocation.
Self-service access to backup to retrieve old files etc. from /project1
Summary of storage available
How to check quotas / usage
Optimizing large files on /project for parallel cluster jobs (advanced HPC users)
10/32
BioHPC Cluster
UTSW private cloud
SSH Web SAMBA FTP
Lamella Cloud Storage No Yes No No
Lamella Cloud Gateway No Yes Yes* Yes*
BioHPC File Exchange No Yes No No
* Do not have access to lamella local cloud storage
Initial Setup for Lamella Gateway
11/32
* Mount External Storage
— Setting up Lamella to access BioHPC cluster (home, project, and work space)
lysosome username password home project
For home leave blank
For lab shared project space: department/lab/shared
BioHPC Endosome/Lysosome
12/32
• Basic File Management view, move, rename, download, upload, delete your files and etc.
• File Synchronization owncloud client
• File Sharing Shared Folder BioHPC File Exchange (https://cloud.biohpc.swmed.edu) BioHPC Lamella Cloud Storage (https://lamella.biohpc.swmed.edu)
• BioHPC manager account A lab manager can share data to non-BioHPC users
New/Upload button:
– Allows you to create new files or upload existing ones from your device
– you can also drop files from Explorer or Finder onto the file view.
– With New button, you can create a text file, folder or download a file from the provided URL.
Search/Settings:
New/Upload button:
– Select multiple files can e deleted or downloaded as ZIP
Viewing Files
– Can display text files, PDFs, and image files
– If displaying of a file is not possible, a download process will start
Moving files
Lamella Cloud Storage
File Synchronization – know your data
Before setting up file synchronization:
1) Estimate the total size of your data if the total size of your data is always < 100GB, you may directly
synchronize your data to lamella, otherwise use your BioHPC project directory.
2) Estimate the total free space on your local disk if the total size of your data is always < free space on your local disk,
you may synchronize everything from lamella, otherwise chose what to sync.
File Synchronization – use synchronization software
16/32
connect to lamella.biohpc.swmed.edu -> username/password
total data size < local disk space
total data size > local disk space
Total data size = actual file stored in lamella locally (up to 100 GB) + mounted folders (up to 10 TB)
Anything your deleted from your PC will be removed from BioHPC cloud storage server as well.
* File Synchronization
File Synchronization
Multiple sync folders
- Skip folder configuration & directly add folder Sync Connection from HOME panel.
19/32
If your are using a BioHPC thin client or workstation you have direct access to BioHPC storage.
File Transfer
Accessing BioHPC Storage Directly from thin client, workstation
20/32
Check ‘Connect using different credentials’ Enter your BioHPC username and password when prompted.
File Transfer
Accessing BioHPC Storage Directly from Mac OSX via Samba
21/32
Folder is: smb://lamella.biohpc.swmed.edu/username home smb://lamella.biohpc.swmed.edu/project/department/group project smb://lamella.biohpc.swmed.edu/work/department/group work
File Transfer
File Transfer
23/32
1) Increase the maximum size of WebDav downloads to 4GB
regedit -> HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WebClient\Parameters\FileSizelimitinBytes
By default Windows limits the maximum size of WebDav downloads to ~50 MB
File Transfer
Not recommended but still useful when you need direct access to the local
storage of BioHPC Lamella or File Exchange system from your PC
24/32
Start -> Computer -> Map Network Drive
* You could mount BioHPC File Exchange storage to your PC in a similar way .
(https//cloud.biohpc.swmed.edu/remote.php/webdav)
25/32
File Transfer
Cyberduck – A Client support FTP, WebDAV for both Mac and Windows
• Download and install Cyberduck from https://portal.biohpc.swmed.edu/content/software/
• Once installed into your 'Applications' folder, start Cyberduck, choose 'Open Connection' and configure as in the image below:
Accessing your Files from Mac OSX via WebDAV
Lamella Cloud Storage (Samba)
400MB/s (10GbE) 100MB/s (1GbE)
Lamella Cloud Storage (FTP )
300 MB - 1.4 GB/s
• Upload and download speed depends on your actual network connection • We don’t set a limitation on max concurrent sessions
File Transfer
File Sharing
• Sharing data inside your department /work/department_name/shared /project/department_name/shared
• Sharing data inside your group /work/department_name/group_name/shared /project/department_name/group_name/shared
If you have a workstation connected to our BioHPC cloud or you know how to use
command line to connect to BioHPC cloud, you can share your data on cluster directly.
Shared folders had already been set up at different levels
28/32
File Sharing
https://lamella.biohpc.swmed.edu
Share with link Share with BioHPC user or group (Email notification)
29/32
File Sharing
Lamella v.s. File Exchange
• lamella.biohpc.swmed.edu • Accessible from internet • 50 GB local storage
• Files stored must not contain PHI or other privacy sensitive information without prior approval.
BioHPC Lamella Cloud Storage
• cloud.biohpc.swmed.edu • Accessible from UTSW network only • 100 GB local storage space • Accessible to BioHPC cluster
31/32
A lab manager ( BioHPC manager account ) sharing data to non-BioHPC users
Lysosome
Lab Manager Lab manger account : only provide storage space under
/project/department/group/manager_name
Lab Facilities, e.g.: scanning EM
images
32/32
A lab manager ( BioHPC user ) sharing data to non-BioHPC users
Lysosome
A lab manager sharing data to non-BioHPC users
33/32
Step 1: mount manager’s home directory to your lab facility to save data
Support PC, Mac and Linux/Unix
A lab manager sharing data to non-BioHPC users
34/32
Step 2: Sharing data with non BioHPC users with a link password protection (recommended) set up an expiration date (recommended) email notification
A lab manager sharing data to non-BioHPC users
35/32
Step 3: non BioHPC users download data by providing correct password
For More Information
• Email the ticket system: biohpc-help@utsouthwestern.edu