Mobycast

Mobycast.fm
Mobycast

A Podcast About Cloud Native Software Development, AWS, and Distributed Systems

  1. 15/04/2020

    Replay of Ep 43 - The Birth of NoSQL and DynamoDb – Part 5

    Show Details Jon Christensen and Chris Hickman of Kelsus and Rich Staats of Secret Stache conclude their series on the birth of NoSQL and DynamoDB. They compare the NoSQL database, Leviathan, created by Chris’s startup in the late 1990s to today’s DynamoDB. A lot of things haven’t changed, even though technology has evolved. It’s cyclical. There are patterns and problems that continue to dominate.    Some of the highlights of the show include: Reason for Creation of NoSQL Database: How to scale database with Internet-scale applications to have a virtual pool of infinite storage that can be scaled outMain Architecture Components of Leviathan:API clientUpdate distributor (UD)Base server (storage node)Shepherd (housekeeping management system)  Additional core components included smart IP and storage abstraction layer (SAL)Leviathan mostly used C code and minimal Java code to support usersBig difference between DynamoDB and Leviathan is request router and partition metadata system living on the server vs. living on the edgeLeviathan was a closed system with an instance for every network or data center; not designed to run as a software as a service, like DynamoDBLeviathan was strongly consistent, unlike DynamoDB’s eventually consistent modelDefinition and Different Types of TransactionsShepherd was used to identify and address consistency, synchronous, and timing issues Rather than using a file system, Leviathan used relational databases Links and Resources DynamoDB Microsoft SQL Oracle DB AWS IoT Greengrass Kelsus Secret Stache Media   Quotes: “We had the same kind of problems that DynamoDB had - how do you scale your database dealing with Internet-scale applications and have this virtual pool of infinite storage that can be scaled out.” Chris Hickman   “This system and this technology went through many iterations.” Chris Hickman   “You can’t have a 100% consistent state across everything. It’s just impossible. How do you do the right thing?” Chris Hickman   “The big difference between DynamoDB and Leviathan...is the request router and partition metadata system living on the server vs. living out at the edge.” Jon Christen

    43min
  2. 08/04/2020

    Replay of Ep 42 - The Birth of NoSQL and DynamoDb – Part 4

    Show Details What’s under the hood of Amazon’s DynamoDB? Jon Christensen and Chris Hickman of Kelsus continue their discussion on DynamoDB, specifically about it’s architecture and components. They utilize a presentation from re:Invent titled, Amazon DynamoDB Under the Hood: How we built a hyper-scale database.    Some of the highlights of the show include: Partition keys and global secondary indexes determine how data is partitioned across a storage node; allows you to scale out, instead of upUnderstand how a database is built to make architecture/component definitions less abstractDynamoDB has four components:          1.   Request Router: Frontline service that receives and handles requests          2.   Storage Node: Services responsible for persisting and retrieving data          3.   Partition Metadata System: Keeps track of where data is located          4.   Auto Admin: Handles housekeeping aspects to manage system What level of uptime availability do you want to guarantee?Replication: Strongly Consistent vs. Eventually ConsistentWalkthrough of Workflow: What happens when, what does it mean when…DynamoDB architecture and components are designed to improve speed and scalabilitySplit Partitions: Longer times that your database is up and the more data you put into it, the more likely you’re going to get a hot partition or partitions that are too big Links and Resources DynamoDB re:Invent Amazon DynamoDB Under the Hood: How we built a hyper-scale database Paxos Algorithm Amazon S3 Amazon Relational Database Service (RDS) MongoDB JSON Kelsus Secret Stache Media Quotes: “Keep in mind that data is partitioned across storage node, and that’s a key feature of being able to scale out, as opposed to scaling up.” Jon Christensen “Amazon was opening up the kimono...how DynamoDB has been architected and constructed and how it works.” Chris Hickman “Managed Service - they get to decide how it’s architected...because they also have to keep it up and live up to their SLA.” Chris Hickman “The longer the time that your database is up and the more data you put into it, the more likely that you’re going to get a hot partition or partitions are just going to get too big.” Chris Hickman

    41min
  3. 01/04/2020

    Replay of Ep 41 - The Birth of NoSQL and DynamoDb – Part 3

    Show Details Jon Christensen and Chris Hickman of Kelsus and Rich Staats of Secret Stache continue their discussion on the birth of NoSQL and DynamoDB. They examine DynamoDB’s architecture and popularity as a solution for Internet-scale databases.  Some of the highlights of the show include: Challenges, evolution, and reasons associated with Internet-scale dataDynamoDB has been around a long time, but people are finally using itDynamoDB and MongoDB are document or key value stores that offer scalability and event-driven programming to reduce complexityTechniques for keeping NoSQL database’s replicated data in syncImportance of indexes to understand query patternsDynamoDB’s Table Concept: Collection of documents/key value items; must have partition key to uniquely identify items in table and determine data distributionSort keys create indexes (i.e. global/local secondary index) to support items within partitioning Query a DynamoDB database based on what’s being stored and using keys; conduct analysis on queries to determine their effectiveness Links and Resources AWS re:Invent DynamoDB NoSQL MongoDB Groupon JSON PostgreSQL Kelsus Secret Stache Media Quotes: “Kind of what drove this evolution from SQL to NoSQL - realizing that the constraints were now different, the economics of the resources that were being used.” Chris Hickman “People are realizing that Dynamo is not an ugly stepchild.” Jon Christensen “Event-driven programming...it’s very popular, and it’s going to become even more popular.” Chris Hickman End Song Benirrás Nights by Roy England ft. Dovetracks

    30min
  4. 25/03/2020

    Replay of Ep 40 - The Birth of NoSQL and DynamoDb – Part 2

    Show Details Jon Christensen and Rich Staats learn about Chris Hickman’s first venture-backed startup (circa 1998) and its goal to build a database for Internet-scale applications. His story highlights what software is all about – history repeating itself because technology/software is meant to solve problems via new tools, techniques, and bigger challenges at bigger scales. Some of the highlights of the show include: Why Chris left Microsoft and how much it cost him; yet, he has no regretsChris’s concept addressed how to build a scalable database layer; how to partition, chart, and cluster; and how to make it highly available and a completely scale-out architectureChris couldn’t use the code he had created for it while at Microsoft; but from that, he  learned what he wouldn’t do againChris let the file system be the database at Microsoft, and the project was named, Internet File Store (IFS); it used backend code and was similar to S3Chris named his startup Viathan; had to do copyright, trademark, and domain name searchesData for the Microsoft project could be stored in files/XML documents; Viathan took a different approach and used relational databases instead of a file systemCompanies experienced problems at the beginning of the Internet; rest of ecosystem wasn’t developed and there weren’t enough people needing Internet solutions yetViathan went through several iterations that led to patents being issued and being considered as Prior artViathan’s technology couldn’t just be plugged in and turned on, applications had to be modified – a tough sellChris did groundbreaking work for what would become DynamoDB Links and Resources AWS DynamoDB AWS re:Invent 2018 – Keynote with Werner Vogels re:Invent DeepRacer JSON Moby Dick MongoDB Acid Compliance Prior Art Kelsus Secret Stache Media

    33min
  5. Replay of Ep 39 -  The Birth of NoSQL and DynamoDB

    18/03/2020

    Replay of Ep 39 - The Birth of NoSQL and DynamoDB

    Chris Hickman and Jon Christensen of Kelsus and Rich Staats from Secret Stache offer a history lesson on the unique challenges of data at “Internet scale” that gave birth to NoSQL and DynamoDB. How did AWS get to where it is with DynamoDB? And, what is AWS doing now?  Some of the highlights of the show include: Werner’s Worst day at Amazon: Database system crashes during Super Saver ShippingAmazon strives to prevent problems that it knows will happen again by realizing relational database management systems aren’t built/designed for the Internet/CloudInternet: Scale up vs. scale out via databases or servers; statefulness of databases prevents easy scalabilityNeed sharding and partitioning of data to have clusters that can be scaled up individuallyAmazon’s Aha Moment: Realization that 90% of data accessed was simplistic, rather than relational; same thing happened at Microsoft - recall the Internet Tidal Wave memo?Challenge of building applications using CGI bin when Internet was brand newSolution: Build your own Internet database; optimize for scalability Links AWS re:Invent DynamoDB NoSQL AWS re:Invent 2018 - Keynote with Andy Jassy AWS re:Invent 2018 - Keynote with Werner Vogels Oracle Database Bill Gates’ Internet Tidal Wave CGI Bin Kelsus Secret Stache Media End Song Whisper in a Dream by Uskmatu More Info We'd love to hear from you! You can reach us at: Web: https://mobycast.fmVoicemail: 844-818-0993Email: ask@mobycast.fmTwitter: https://twitter.com/hashtag/mobycastReddit: https://reddit.com/r/mobycast

    33min
  6. Replay of Ep 14. Stop Worrying About Cloud Lock-in

    11/03/2020

    Replay of Ep 14. Stop Worrying About Cloud Lock-in

    Original Show Notes:At the recent Gluecon event, a popular topic centered around how to prevent Cloud Lock-in. Chris Hickman and Jon Christensen of Kelsus and Rich Staats from Secret Stache discuss why you your time is better spent focusing on one cloud provider. If/when Cloud Lock-in becomes an issue, you will have the resources to deal with it. Some of the highlights of the show include: AWS Fargate is ‘serverless ECS’. You don’t need to manage your own cluster nodes. This sounds great, but we’ve found the overhead of managing your own cluster to be minimal. Fargate is more expensive than ECS, and you have greater control if you manage your own cluster.Cloud lock-in was a huge concern among people at Gluecon 2018. People from large companies talked about ‘being burned’ in the past with vendor lock-in. The likely risks are (1) price gouging and (2) vendors going out of business.Cloud allows people to deploy faster and more cheaply than running their own hardware, as long as you don’t have huge scale. Few businesses get large enough to need their own data center on-prem to save money.Small and startup companies often start off in the Cloud. Big companies often have their own data centers and they are now migrating to the Cloud.AWS does allow you to run their software in your own data center, but this ties you to AWS.There is huge complication and risk to architecting a system to run in multiple cloud environments, and it almost certainly wouldn’t run optimally in all clouds.We think the risk of AWS hiking prices drastically, or going out of business, is essentially zero.If you were building a microservice-based multi-cloud system, some of the difficulties include: Which cloud hosts the database? How do I spread my services across 2 clouds? What about latency between cloud providers networks? How do I maintain security? How do I staff people who are experts at operating in both clouds?It’s clear that lock-in is a real fear for many companies, regardless of our opinion that it shouldn’t be such a concern.Jon thinks the fear of lock-in may drive cloud providers toward standardization; Chris thinks AWS doesn’t have a compelling reason to standardize since they’re the industry leader.Our advice: as a small or medium size company, don’t worry about cloud lock in. If you get big enough that it’s really a concern, we recommend building abstractions for the provider-specific parts of your system, and having a backup of your system ready to run in a 2nd cloud provider, but don’t try to run them concurrently.Links and Resources KelsusSecret Stache MediaAWS Fargatere:InventGlueconKubernetes

    27min
5
de 5
22 avaliações

Sobre

A Podcast About Cloud Native Software Development, AWS, and Distributed Systems

Para ouvir episódios explícitos, inicie sessão.

Fique por dentro deste podcast

Inicie sessão ou crie uma conta para seguir podcasts, salvar episódios e receber as atualizações mais recentes.

Selecionar um país ou região

África, Oriente Médio e Índia

Ásia‑Pacífico

Europa

América Latina e Caribe

Estados Unidos e Canadá