Serverless Logging with AWS Lambda and the Elastic Stack

SERVERLESS LOGGING WITH ELKEdoardo Paolo Scalafiotti - Software Engineer @ Sainsbury’s

About me

Go, NodeJS

SPAs, WebComponents

Material Design

Serverless Architecture

Microservices

Machine Learning

FabLab, Meetups

AgendaOVERVIEWWHY SERVERLESS?CHALLENGESUSE CASESDEMOQ & A

WHAT IS SERVERLESS?

TL;DRFaaS is about running back end code without managing your own server

systems or your own server applications

applications that run in stateless compute containers that are event-

triggered, ephemeral and fully managed by a 3rd

party

● 2 overlapping areas: BaaS and PaaS● run code without provisioning or

managing servers● horizontal scaling is completely

automatic, elastic, and managed by the provider

● responds to triggers

WHY SHOULD WE CARE?

serverless is

HOT!

Typical servers in business and enterprise data centers deliver between 5 and 15 percent of their maximum computing

output on average over the course of the year

● Reduced operational cost● Good with irregular, high spike

traffic that cannot be handled by auto-scaling

● Promotes best coding practises

WHAT ARE THE CHALLENGES?

TL;DRnot for the faint-hearted... at the moment

● Ugly hacks for keeping the function alive

● Vendor lock-in● Unit testing is ok, integration... another

story● Repetition of logic● Concurrent limit● Logging, Monitoring and Debugging

USE CASES

USE WISELYFaaS, containers and VMs are not

enemies

FAAS is seen as a better choice for event-driven apps with few event types

per application component,while CONTAINERS are seen as a better

choice for synchronous-request driven components with many entry points

VM Containers ServerlessConfigure machines, storage,

networking and OS

Run servers, configure

applications and control scaling

Run my code when it’s needed

Use cases:● Real-time file processing (ingestion)● Real-time stream processing (mkt.

data)● Extract, Transform, Load● IoT Backends● Mobile Backends● RESTful APIs

● Open Source alternative: IBM’s OpenWhisk

● API Orchestration: serverless.com and claudia.js

● Multiple PL support: Apex● Logging and monitoring: ELK

LET’S GET TO THE CODE...

THE MISSION...is to monitor a set of Lambda-based

functions that generates Game Of Thrones plot points

DEMO

github.com/edoardo849/lot