Upload
jurgis-pasukonis
View
144
Download
6
Tags:
Embed Size (px)
Citation preview
Plan1. About this meetup – why and what
2. TRAFI case◦ Architecture overview
◦ Azure vs. AWS
◦ Queues and data streams (AWS Kinesis)
◦ Google BigQuery
Why meetup?Community is good!
◦ Share knowledge
◦ Similar problems – someone else maybe solved it
◦ Different approaches – widens your view
◦ It’s fun
What is “backend developer”?BEFORE NOW
ASP.NET
SQL
Mobile
API API
Load balancer
Queues
Cache
Logs
Metrics SQL NoSQL
Web JS
Worker
IAPI
Lots to learnCan build a lot very easily… if you know how
Challenges◦ Mobile, cloud, service-oriented architecture
◦ New tools every day (libraries, frameworks, cloud services)
◦ Agile, cross-functional teams
◦ No longer “C# devs”, “Java devs” and “SQL devs” – need to understand bigger picture
Meetup focusArchitecture & patterns
◦ How to build cloud (distributed) solutions
Tools & technologies
◦ Know what’s out there (right tool for the right job)
Possible topicsArchitecture & patterns
◦ Service-oriented, queues, micro services◦ Async, reactive programming◦ Functional programming◦ Domain driven design, hexagonal architecture
Languages◦ C#, Scala, Go, Java8, C++11, Ruby, Node.js, …
NoSQL◦ MongoDB, Redis, Firebase
Big data◦ Hadoop, Spark, BigQuery◦ Machine learning
Cloud services◦ Amazon Web Services (38 services!)◦ Microsoft Azure◦ Google Cloud
Anything you’ve found out, you wished someone told you!
TRAFIUse case
◦ Android, iOS, Web apps
◦ REST API (~100 reqs/sec)
◦ Memory & Compute intensive
◦ External data integrations
◦ Background jobs
Code◦ C#, ASP.NET MVC
◦ Some Java
◦ Some Python
TRAFI architectureMobile
.NET API .NET API
Elastic Beanstalk(load balancer)
Data streaming(Kinesis)
ElastiCache
Logentries(logs)
Librato(metrics)
MS SQL BigQuery
Azure Website
Background jobs(Win VM)
Internal Java API(Linux VM)
Infrastructure◦ AWS Elastic Beanstalk
◦ AWS EC2 instances (c3.4xlarge)
◦ Azure websites
◦ Azure worker roles
Storage◦ MS SQL server
◦ AWS Kinesis
◦ AWS ElastiCache
◦ Google BigQuery
Metrics & Logs◦ Logentries
◦ Librato
Lesson #1: Azure vs AWSMigrated .NET API
◦ Azure: Web roles + Azure SQL
◦ AWS: Elastic Beanstalk + RDS SQL
Advantages◦ 2x faster CPU performance!
◦ Faster deployment (also “.NET-friendly”)
◦ More flexible load balancer, auto scaling
◦ Better metrics and logs
◦ SQL – faster, easier to manage, better metrics
Disadvantages◦ No “staged rollout”, only swap
Lesson #2: use messagingUse case: streaming data from external sources
Don’t reinvent the wheel – it’s not as easy as it looks!
External integration
API 1
API 2GPS data DB
Background
External integration
API 1
API 2GPS data
Background
Stream
Lesson #2: use messagingQueues
◦ AWS SQS
◦ Azure storage queues
◦ RabbitMQ
Data streams (publish/subscribe)◦ AWS Kinesis
◦ Azure Service bus
◦ Kafka
Queue
Worker 1
Worker 2B
A
B
Stream
Listener 1
Listener 2B
A
B
A
A
B
A
Lesson #3: storing logsUse case: store insert-only data for later analysis
◦ Web server request logs
◦ App event logs
◦ GPS logs…
Storing logs attempt 1Traditional SQL
◦ Good for small amount of data (<10 GB)
◦ Limited insert capacity
◦ Slow queries
◦ Expensive ($1.00 /GB/month)
Storing logs attempt 2Azure table storage (NoSQL)
◦ Cheap ($0.07 /GB/month)
◦ Unlimited data (partitioning)
◦ Only simple queries, slow
Storing logs attempt 3Google BigQuery (columnar store)
◦ Very Cheap ($0.02 /GB/month)*
◦ Unlimited data (partitioning)
◦ Fast advanced SQL queries!
◦ Structured data
Open questionsWebsites
◦ Pretty nice in Azure – easy deployment, cheap
Background jobs◦ No “worker roles” in AWS
◦ Running as “Windows Services” in custom EC2 VM
◦ Docker?