40
HU-Stack: Eine cloud-basierte Plattform als flexible Service- Infrastruktur Malte Dreyer

HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

Embed Size (px)

Citation preview

Page 1: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

HU-Stack: Eine cloud-basierte Plattform als flexible Service-Infrastruktur

Malte Dreyer

Page 2: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

Profil / Struktur

Computer- und Medienservice (CMS)

Verwaltung (zentral)

• Ressourcen-Mgmt.• Campus-Mgmt.• Verwaltungs PCs

Hardware / Software• Endgeräte (~3.000)• SW-Lizenz-Mgmt.• Benutzerberatung

Digitale Medien• Medien-Mgmt.• Lernumgebungen• Audio-/Video-Technik

Basis-Infrastruktur• Server• Storage• Netze

„Nachwuchsgruppen“„Nachwuchsgruppen“

Page 3: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

Profil / Struktur

Computer- und Medienservice (CMS)

Verwaltung (zentral)

• Ressourcen-Mgmt.• Campus-Mgmt.• Verwaltungs PCs

Hardware / Software• Endgeräte (~3.000)• SW-Lizenz-Mgmt.• Benutzerberatung

Digitale Medien• Medien-Mgmt.• Lernumgebungen• Audio-/Video-Technik

Basis-Infrastruktur• Server• Storage• Netze

„Nachwuchsgruppen“„Nachwuchsgruppen“„Nachwuchsgruppen“

Projekte und Plattformen(auch Forschungsdatenmanagement)

AG ePublishing(gemeinsam mit UB)

Page 4: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

Basisinfrastruktur

10.09.2013

Computer- und Medienservice4

Page 5: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

Zusammenarbeit mit den Einrichtungen

Servicetiefe des CMS

Plattformen

Dienste

Basis-Infrastruktur

Arbeitsplätze

Computer-und

Medien-Service

Computer-und

Medien-Service

Inst. 2

Computer-und

Medien-Service

Inst. 3

Computer-und

Medien-Service

ZentraleVerwaltung

Inst. 1

Page 6: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

Breites Diensteportfolio

Identity Management Service

Image Editing

Mail Service

Mailing Lists

Map Service

Moodle

Printing Service

Public Computers

Publication Repository

Resource Management SW

Scanning Service

Shibboleth

Software Service

Statistics SW Service

Survey

Terminalserver Service Service

Professional Training

Trouble Ticket Systems

User Certificates

End User Support

Virtual Private Networks

Video Conference Service

Video Service

WLAN

Web Content Management

Web Files

Web Hosting

Website Management

Wiki and Blog Farms

Windows Domains

ePublishing

ADS

AFS

Accounts

Anti Virus Software

Audio / Video Equipment

Backup-Service

Calendar Service

Campus Management SW

Chat Service

Compute Clusters

Database Service

Managed Desktops

File Exchange

File Service

Graphics

Hardware Service

Page 7: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

Breites Diensteportfolio

…aber noch nicht breit genug -> Forschungsdaten

Identity Management Service

Image Editing

Mail Service

Mailing Lists

Map Service

Moodle

Printing Service

Public Computers

Publication Repository

Resource Management SW

Scanning Service

Shibboleth

Software Service

Statistics SW Service

Survey

Terminalserver Service Service

Professional Training

Trouble Ticket Systems

User Certificates

End User Support

Virtual Private Networks

Video Conference Service

Video Service

WLAN

Web Content Management

Web Files

Web Hosting

Website Management

Wiki and Blog Farms

Windows Domains

ePublishing

ADS

AFS

Accounts

Anti Virus Software

Audio / Video Equipment

Backup-Service

Calendar Service

Campus Management SW

Chat Service

Compute Clusters

Database Service

Managed Desktops

File Exchange

File Service

Graphics

Hardware Service

Page 8: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

Biographical Sketch(es), has been revised to rename the “Publications”

section to “Products” and amend terminology and instructions accordingly.

(c) Products

A list of: (i) up to five products most closely related to the proposed

project; and (ii) up to five other significant products, whether or not

related to the proposed project. Acceptable products must be citable

and accessible including but not limited to publications, data sets,

software, patents, and copyrights. Unacceptable products are

unpublished documents not yet submitted for publication, invited

lectures, and additional lists of products. Only the list of 10 will be used

in the review of the proposal.

Each product must include full citation information including (where

applicable and practicable) names of all authors, date of publication or

release, title, title of enclosing work such as journal or book, volume,

issue, pages, website and Uniform Resource Locator (URL) or other

Persistent Identifier.

National Science Foundation (NSF)

Biographical Sketch(es)

NSF 13-1 January 2013

http://www.nsf.gov/pubs/policydocs/pappguide/nsf13001/gpg_2.jsp#IIC2d

Page 9: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

1. Recommended open access to scholarly papers of

publicly funded research

2. Recommended open access to all digital outputs of

publicly funded research

3. Mandated open access to scholarly papers of publicly

funded research

4. Mandated open access to all digital outputs of

publicly funded research

5. Enforced, mandated open access to scholarly papers

of publicly funded research

6. Enforced, mandated open access to all digital

outputs of publicly funded research

Page 10: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

By Muriel (originally posted to Flickr as Muon Barrel) [CC-BY-SA-2.0 (http://creativecommons.org/licenses/by-

sa/2.0)], via Wikimedia Commons

Page 11: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

By Hannes Grobe (Hgrobe 06:16, 26 April 2006 (UTC)), Alfred Wegener Institute for Polar and Marine Research

(Own work) [CC-BY-SA-2.5 (http://creativecommons.org/licenses/by-sa/2.5)], via Wikimedia Commons

Page 12: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

Small Data

„Long Tail of Research Data“

Source: Ferguson et al. (2014): Big data from small data: data-sharing in the

'long tail' of neuroscience. DOI: 10.1038/nn.3838

Science 11 February 2011: Vol. 331 no. 6018 pp. 692-693 DOI: 10.1126/science.331.6018.692

The 2011 survey by Science, found that 48.3% of respondents were working with datasets that were less than 1GB in size and over half of those polled store their data only in their laboratories.

Page 13: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

Simukovic, Elena; Kindling, Maxi; Schirmbacher, Peter (2013): Umfrage zum Umgang mit digitalen Forschungsdaten an der

Humboldt-Universität zu Berlin. Umfragebericht, Version 1.0. URN: urn:nbn:de:kobv:11-100213001n=457

Page 14: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

https://www.cms.hu-berlin.de/de/ueberblick/projekte/dataman/home

Page 15: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

Anpassungen und Abbildung

von Szenarien

z. B. Medien-Repositorien-Farm

Gemeinsame Projekte und

Software-Entwicklung

z. B. Laudatio Corpus Repositorium

Projekte in enger Zusammenarbeit mit den Instituten

(gemeinsame Drittmitteleinwerbung)

Page 16: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

80 Individuelle Projekte

80 Installationen

80 Datenbanken

80 Konfigurationen

Individuelle Software

Fedora Repository

CakePHP

Elastic Search

PID-Server (Handle)

jQuery / Bootstrap

Apache Xerces

+RedMine for Tickets

Aus Sicht des Datenzentrums

... Nutzer erwarten dauerhaften Betrieb …

Page 17: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

80 Individual Projets

80 Installations

80 Databases

80 Configurations

Individual Software

Fedora Repository

CakePHP

Elastic Search

PID-Server (Handle)

jQuery / Bootstrap

Apache Xerces

+RedMine for Tickets

Aus Sicht des Datenzentrums

... Pflegeaufwand …

Änderungen und Erweiterungen

(Maßkonfektion)

Überwiegende Eigenentwicklung(Maßgeschneidert)

Page 18: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

Selbstadministration / Selbstbedienung

By Paul Hudson from United Kingdom (11" MacBook Air) [CC BY 2.0

(http://creativecommons.org/licenses/by/2.0)], via Wikimedia Commons

By HLundgaard (Own work) [CC BY-SA 3.0

(http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons

Page 19: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

By Sae1962 (Own work) [CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons

Page 20: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

Flexibilisierung der Infrastruktur

XaaS „Anything as a Service“

Selbstbedienungsfunktionen (cloud)

IaaSInfrastructure

PaaSPlatforms

SaaSSoftware /

Applications

HardwareServer / Storage /

Network

HardwareServer / Storage /

Network

HardwareServer / Storage /

Network

IaaSInfrastructure

IaaSInfrastructure

PaaSPlatforms

Page 21: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

Cloud computing is a model for enabling ubiquitous,

convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released withminimal management effort or service provider interaction.

The NIST Definition of Cloud Computing

(US-National Institute of Standards and Technology SP 800-145)

Page 22: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

Automatisierung /

Mehr Softwareentwicklung und –Technik

By Mixabest [CC BY-SA 3.0 (http://creativecommons.org/licenses/by-

sa/3.0), via Wikimedia Commons

Page 23: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

Das Virtuelle Rechenzentrum

HEAT

my_port01:

type: OS::Neutron::Port

properties:

admin_state_up: true

name: Port01

network_id: { get_resource: my_first_network }

my_server07:

type: OS::Nova::Server

properties:

name: seaf07

key_name: { get_param: keypair_name }

flavor: { get_param: machine_flavor }

image: { get_param: image_id }

networks: [{ "port": { get_resource: my_port07 }, …}]

user_data: |

#!/bin/bash -v

….

cinder_volume07:

type: OS::Cinder::Volume

properties:

size: { get_param: volume_size }

volume_attachment07:

type: OS::Cinder::VolumeAttachment

properties:

volume_id: { get_resource: cinder_volume07 }

instance_uuid: { get_resource: my_server07 }

mountpoint: /dev/vdb

ANSIBLE Playbook

# update packet cache

- hosts: all

tasks:

- apt:

- name: apply common configuration to all nodes

hosts: all

roles:

- common

- name: deploy and configure memcached server

hosts: memcached

roles:

- memcached

- name: deploy MySQL and configure the databases

hosts: db

roles:

- db

- name: deploy seafile and apply configuration

hosts: nodes

roles:

- nodes

- name: deploy loadbalancer

hosts: loadbalancer

roles:

- loadbalancer

tags:

- loadbalancer

Page 24: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

Computer- und Medienservice

Page 25: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

Computer- und Medienservice

Page 26: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

Horizon

Dashboard

Nova

Compute

Neutron

Network

Swift

Object Storage

Glance

Image Mgmt

Heat

Orchestration

Ceilometer

Metering

Keystone

Identity Mgmt

Cinder

Block Storage

Trove

Database Mgmt

Sahara

Hadoop

OpenStack Komponenten

Page 27: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

• Preise [TEUR]

2016 2017 2018

WDM (Licht) 200 200

Linecards 10 Gbps x20

200

MLX (Router) Chassis 8x

175 175

Linecards(26) 10Gbps x8

286 286

Summe 661 661 200

Page 28: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms
Page 29: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

Horizon

Dashboard

Nova

Compute

Neutron

Network

Swift

Object Storage

Glance

Image Mgmt

Heat

Orchestration

Ceilometer

Metering

Keystone

Identity Mgmt

Cinder

Block Storage

Trove

Database Mgmt

Sahara

Hadoop

Technische Komponenten der Cloud / Organisatorische Einbettung

Page 30: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

Horizon

Dashboard

Nova

Compute

Neutron

Network

Swift

Object Storage

Glance

Image Mgmt

Heat

Orchestration

Ceilometer

Metering

Keystone

Identity Mgmt

Cinder

Block Storage

Trove

Database Mgmt

Sahara

Hadoop

Technische Komponenten der Cloud / Organisatorische Einbettung

Page 31: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

Horizon

Dashboard

Nova

Compute

Neutron

Network

Swift

Object Storage

Glance

Image Mgmt

Heat

Orchestration

Ceilometer

Metering

Keystone

Identity Mgmt

Cinder

Block Storage

Trove

Database Mgmt

Sahara

Hadoop

Technische Komponenten der Cloud / Organisatorische Einbettung

Page 32: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

Horizon

Dashboard

Nova

Compute

Neutron

Network

Swift

Object Storage

Glance

Image Mgmt

Heat

Orchestration

Ceilometer

Metering

Keystone

Identity Mgmt

Cinder

Block Storage

Trove

Database Mgmt

Sahara

Hadoop

Technische Komponenten der Cloud / Organisatorische Einbettung

Page 33: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

Horizon

Dashboard

Nova

Compute

Neutron

Network

Swift

Object Storage

Glance

Image Mgmt

Heat

Orchestration

Ceilometer

Metering

Keystone

Identity Mgmt

Cinder

Block Storage

Trove

Database Mgmt

Sahara

Hadoop

Technische Komponenten der Cloud / Organisatorische Einbettung

Page 34: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

Horizon

Dashboard

Nova

Compute

Neutron

Network

Swift

Object Storage

Glance

Image Mgmt

Heat

Orchestration

Ceilometer

Metering

Keystone

Identity Mgmt

Cinder

Block Storage

Trove

Database Mgmt

Sahara

Hadoop

Technische Komponenten der Cloud / Organisatorische Einbettung

Page 35: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

Horizon

Dashboard

Nova

Compute

Neutron

Network

Swift

Object Storage

Glance

Image Mgmt

Heat

Orchestration

Ceilometer

Metering

Keystone

Identity Mgmt

Cinder

Block Storage

Trove

Database Mgmt

Sahara

Hadoop

Technische Komponenten der Cloud / Organisatorische Einbettung

Page 36: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

Horizon

Dashboard

Nova

Compute

Neutron

Network

Swift

Object Storage

Glance

Image Mgmt

Heat

Orchestration

Ceilometer

Metering

Keystone

Identity Mgmt

Cinder

Block Storage

Trove

Database Mgmt

Sahara

Hadoop

Technische Komponenten der Cloud / Organisatorische Einbettung

Page 37: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

Mein Virtuelles Rechenzentrum

ASAP

HEAT

my_port01:

type: OS::Neutron::Port

properties:

admin_state_up: true

name: Port01

network_id: { get_resource: my_first_network }

my_server07:

type: OS::Nova::Server

properties:

name: seaf07

key_name: { get_param: keypair_name }

flavor: { get_param: machine_flavor }

image: { get_param: image_id }

networks: [{ "port": { get_resource: my_port07 }, …}]

user_data: |

#!/bin/bash -v

….

cinder_volume07:

type: OS::Cinder::Volume

properties:

size: { get_param: volume_size }

volume_attachment07:

type: OS::Cinder::VolumeAttachment

properties:

volume_id: { get_resource: cinder_volume07 }

instance_uuid: { get_resource: my_server07 }

mountpoint: /dev/vdb

ANSIBLE Playbook

# update packet cache

- hosts: all

tasks:

- apt:

- name: apply common configuration to all nodes

hosts: all

roles:

- common

- name: deploy and configure memcached server

hosts: memcached

roles:

- memcached

- name: deploy MySQL and configure the databases

hosts: db

roles:

- db

- name: deploy seafile and apply configuration

hosts: nodes

roles:

- nodes

- name: deploy loadbalancer

hosts: loadbalancer

roles:

- loadbalancer

tags:

- loadbalancer

Page 38: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

..…Danke…..

[email protected]

Page 39: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

Fragen?

[email protected]

Page 40: HU-Stack: Eine cloud-basierte Plattform als flexible ... · Image Editing Mail Service Mailing Lists ... Web Hosting Website Management Wiki and Blog Farms Windows Domains ... Platforms

Computer- und Medienservice

Backup