001. MQs, storages, and dataframes DEbrief

A few hot updates



Apache Geode 1.12.5enterprise edition is known as gemfiregeodistributed storagehas native clients in Java, C#, and C++ (!)JTA compliant transaction supportPinot released 0.9.0Added Segment Merge and RollupRollup is a technique for tree-like groupby
example: city, streets, housesGeneral info about pinotMade by guys from LinkedIn and Uber
has zookeeper as depscolumn-oriented databaseIt's an OLAP tool for real-time analyticsthere are BI tools focused on dashboards and reports
used by analists etcthis is more for data exploration
for de / ds folksNear real-time ingestion from streams (Kafka, Kinesis, and batch ingestion from Hadoop/S3 and the likes)It has built-in UI for SQL edits and general BI for exploration
focus on realtime analyticsYou can connect Pinot to various BI tools such Superset, Tableau, or PowerBI to visualize data in PinotRocketMq 4.9.0 / 4.9.2 Comparison TableBased on ActiveMQDoes not need ZooKeeperHas conecept of strict message orderHas focus on perfect configuration OOTBRich web interfaceSQLite 3.37new STRICT table definition and ANY type
works as cast on writecli client update:multiple connections from the same clientsecurity mode with `-safe`author is well-known as a supporter of flexible typing, have a read https://sqlite.org/flextypegood.html



Ligthning



Superset 1.3.2bugfixesif you never saw what's 1.3.0 has to offer, check it out: they have funnelsalso revised treemap vizBeam 2.34.0NiFi new release (1.15.0)main feature is parameter context inheritanceApache Ratis releaseRaftAirflow 2.2.2bugfixes 🤷‍♀️Nats 2.6.5 recent releasebugfixes



Discussion: Are dataframes necessary?



Kotlin DataFrame



Music by https://t.me/red_hands






Dr. Igor MosyaginData Engineer @ KlarnaIgor identifies himself as a pragmatic engineer with strong academic background. A theoretical physicist by training, he eventually assumed he had enough PhDs and left Academia to work with Data-* related things. As of 2022, Igor works as a Data Platform Engineer at Klarna. On top of that, he’s a huge fan of cephalopods, math rock, and quantum mechanics. He also hates baked carrots so much he decided to mention it in this bioVisit Website (opens in a new tab)Visit Twitter account (opens in a new tab)Visit LinkedIn account (opens in a new tab)Email



Pasha FinkelshteynDeveloper advocate @ JetBrainsHaving 14 years of experience in IT, Pasha went through a fire in water, from technical support to developer, team lead, and data engineer. Now Pasha works as a developer advocate for Data Engineering at JetBrains. He helps develop the Big Data Tools plugin, gives talks on Kotlin and various aspects of data engineering, and work with data. Also, he is the author and maintainer of Kotlin API for Apache Spark.Visit Website (opens in a new tab)Visit Twitter account (opens in a new tab)Visit Facebook account (opens in a new tab)Visit Instagram account (opens in a new tab)Visit LinkedIn account (opens in a new tab)Visit GitHub account (opens in a new tab)Email