Spring One Platform: Culture vs. Code or Culture as Code

Cross posted from my LinkedIn article.

It’s really telling when the first hour of one of the geekiest developer conferences in the industry, Spring One Platform, doesn’t show a single line of code. Instead much of the discussion was about culture and dysfunctions in culture.

Keynoters we heard from in the first hour of the first day include:

Onsi Fakhouri who discussed about the divides between developers and operators.

Cornelia Davis who discussed the gender gap that exists in the software industry these days, and how we can fix this. View the trailer for Debugging the Gender Gap for a great documentary about this topic. (Note: Cornelia actually did present the first lines of code at the conference – 4 lines of her first BASIC program as a middle school student).

Bridget Kromhout who made the point that artificial technical boundaries such as containers won’t fix a broken culture.

Phil Webb who mentioned of the divisions between DBAs and operators and developers in an IT department.

Pointing out cultural dysfunction in IT is nothing new in the industry.   Salesforce made a lot of hay in its earlier days by promising “no software” – just bypass the IT department altogether. That was the beginning of chapter 2 of the “shadow IT” narrative where developers and business users procure their own SaaS and cloud services without asking permission from a centralized corporate IT function.

A funny thing happened in the industry, however. Custom software didn’t go away. It’s now much more pervasive and important to companies. It’s also become a source of differentiation. Bypassing IT didn’t make software development better.  Now, companies are looking to the corporate IT function to better enable their developers to deliver unique business value.

The various technologies and practices discussed at Spring One Platform have a lot to do with mitigating and bridging these company cultural gaps. Onsi’s talk, which you can view an earlier rendition on video, does a nice job of explaining how we might all get along in one happy software collaborative lifecycle.

Now culture can be inscribed into the code that automates the innovation lifecycle.

Geode Summit Takeaways: User Stories, New Architecture, and Futures

Cross posted from The Pivotal Blog.

If you missed the amazing, first Apache Geode Summit held this past March, never fear, all of the slides and video replays of the sessions have been posted for your viewing pleasure.

We can describe this content in just a few words—amazing talks, excellent speakers, deep experience.

And, here are three big things you can learn by checking out the replays:

• #1: Apache Geode Scales Mission Critical Financial Services & Travel Apps
• #2: Geode Powers Modern, Data-Driven Architectures
• #3: Geode Has a Promising Future Ahead

Each of these are 30 minute sessions. So, it is easy to make some time and check them out. Microwaving a bowl of popcorn before viewing is recommended to maximize your enjoyment.

Takeaway #1: Geode Scales Mission-Critical Financial Services & Travel Apps

We heard from a number of production users about their high scale, high performance applications which take advantage of Geode’s distributed in-memory processing.

Combining Stream Processing and In-Memory Data Grids for Near-Real Time Aggregation and Notifications

First, Olivier Mallassi presented how Murex uses Geode in their risk management solution for financial services.  In his case, Apache Storm and Geode are integrated, even collocated, and applied to the fast processing side of a lambda architecture. While Geode is used for immutable events and event logging, one of the most powerful features he covers is continuous queries. Developers can think of these as a distributed, scalable set of in-memory listeners which use data to identify and push event notifications to clients.

Slides can be found here.

Wall St. Derivative Risk Solutions Using Geode

Then we heard from Andre Langevin, currently with Canadian Imperial Bank of Commerce, who has built several such systems using the technology in Apache Geode.  In his talk, he shares a series of insightful architecture whiteboards, explaining how Geode is used as a data  AND compute grid, rolling up financial risk across multiple trading systems.

Slides can be found here.

Design Tradeoffs in Distributed Systems—How Southwest Airlines Uses Geode

Finally, we heard from Brian Dunlap of Southwest Airlines. He explained just how Geode provides real-time intelligence, integrating data from reservation feeds, checkins, kiosks, boarding, gates, flights—basically from everything that moves. After explaining how his system supports critical operations and decisions, regularly crunching out optimizations for over 1,000,000 schedules, he shares many of the lessons learned

Slides can be found here.

Takeaway #2: Geode Powers Modern, Data-Driven Architectures

If the previously mentioned talks didn’t give you a sense of Geode’s fit in data-driven architectures, we also heard from a number of architects on design patterns for Geode in modern applications.

Where Does Geode Fit in Modern System Architectures?

With years of experience using Geode, Eitan Suez explained how developers and architects, like him, get started with Geode and realize that it can be used to solve a lot of problems—that it is basically a distributed hash map. He then presented an approach to using Geode as part of a CQRS (Command Query Responsibility Segregation) architecture.

Slides can be found here.

Architecting Data-Driven, Smarter Cloud Native Apps with Real-Time Decision-Making

Next, we heard from Fred Melo. He presented a compelling explanation and demo of HTAP (Hybrid Transactional / Analytical Processing) architectures, with real-time predictive analytics, featuring Apache Geode.

Slides can be found here.

Large Scale Fraud Detection Using GemFire Integrated with Greenplum

Finally, we heard about a fraud detection case study where Apache Geode was deployed as a pre-processing engine to detect events in streaming data before being loaded into a data warehouse. This included a business-logic processing layer for custom applications above the database. This particular application used the GemFire Connector to Greenplum Database.

Slides can be found here.

Takeaway #3: Geode Has a Promising Future Ahead

In another track of the Summit, many of the committers and contributors of Apache Geode convened to discuss the current and future state of Apache Geode’s feature set.  When you look at the list of capabilities below, you can see that Apache Geode as some very compelling potential features in its future.

• Spring Data GemFire API current & future
• Integration & Future Direction for Spring Cloud Data Flow & Geode
• Redis to Geode Adaptor
• Off-Heap Storage Current and Future Design
• Integration with Spark – Geode reading and writing RDDs

Many of these features are still in their conceptual phase, and the Apache Geode community is seeking people interested to help contribute to make these features better.  We finished up with a great talk by Nitin Lamba who described his journey to becoming an Apache Geode committer: Easy Ways to Become a Contributor to Apache Geode. You can also join the mailing list.

Time to Tailor Your Schedule for Geode Summit – This Weds, March 9th

We’ve just put the final touches on the schedule for Geode Summit. It’s going to be an amazing day with over 100 users and developers of Apache Geode (incubating) meeting for the very first time. We’ll be sharing expertise, networking for jobs, hearing about use cases, solution architectures, future feature designs, and conducting hands-on training.

There are two concurrent tracks running most of the day, some of which have limited space, and some of which can be attended virtually if you aren’t able to be in Palo Alto on March 9.  All the sessions, other than the hands-on lab, are being recorded so you can view them later.

If you haven’t registered yet, there’s still a little space.

There are three main types of sessions at Geode Summit:

Keynotes, Solutions, and Use Cases

This is the main track, available for all attendees, and especially suitable for technology mangers and solution architects.  Here you’ll have a chance to hear what people have done with Geode and GemFire in the past, and get some new ideas of what you can build.

Technology training

This is a 2 hour hands-on training class in the morning for developers who are new to using Apache Geode and GemFire and would like a quick introduction. Space is limited, so be sure to register and add this to your schedule if you want to participate.

Note that Participants must bring their own laptop with the following minimum configuration: Windows/Mac/Linux OS with minimum 8GB RAM Multi-core processor is highly recommended Lab environment and files will require roughly 500MB of space Have Virtual Box installed (we will deliver the environment as a VM with components installed and ready to go)

Contributor Feature Discussions

These are 30 minute deep dives into early / nascent features of Apache Geode.  An Apache Geode Committer will head a short discussion about the current capability of a feature, and then ask for ideas and use cases about how this feature could be used.  The purpose is to help inform the future direction of specific Geode capabilities.

These sessions are streamed virtually for the benefit of the Apache Geode community. You can log in here to participate: https://pivotalcommunity.adobeconnect.com/clubhouse/

Specifically these session, and their broadcast times are:

• Spring Data GemFire API current & future (1300 PST)
• Integration & Future Direction for Spring Cloud Data Flow & Geode (1330 PST)
• Redis to Geode Adaptor (1400 PST)
• Off-Heap Storage Current and Future Design (1430 PST)
• Integration with Spark – Geode reading and writing RDDs (1515 PST)
• Easy Ways to Become a Contributor to Apache Geode (1545 PST)

See you either in person, or virtually at Geode Summit!

Meet Experts and Productive Users of Apache Geode & GemFire at Geode Summit

Something amazing is happening in Palo Alto March 9. It’s the first gathering of over 100 users and developers of the GemFire and Apache Geode (incubating) in-memory data grid technology at Geode Summit.  What were previously dark arts and specialized techniques will be revealed to the open source community. Speakers from a dozen companies that have deployed or are running some of the largest production instances of the technology will share their experience and use cases. There will be hands-on training, and keynotes from industry luminaries about the future of in-memory computing and big data.

Geode was born in the fallout from Enron

The technology that became Apache Geode (incubating) was first invented trying to help a financial services startup scale up to meet new opportunity as a result of the spectacular failure of Enron.  Watch this interview with Richard Lamb, a co-founder of the technology, about genesis of GemFire by Gemstone Systems.

GemFire continued a successful growth path once Gemstone Systems was acquired by VMWare, and then spun out to become a key part of Pivotal.  Last year Pivotal proposed to The Apache Software Foundation creation of the Apache Geode (incubating) community to help foster ongoing development of the technology collaboratively with the open source community. 

An unforgettable lineup of speakers & sessions

 Geode Summit is the first time that committers and new users of Apache Geode can come together, and also meet with long standing users and developers of the commercial GemFire software.

 The morning kicks off with an opening keynote from Justin Erenkrantz, head of compute architecture at Bloomberg. Justin is also well known for his mentoring and contributions to many open source projects in The Apache Software Foundation.

 After his talk, Justin will serve as the moderator for a panel of noted industry pioneers, investors, and startup founders who are building the next generation of technologies for in-memory computing.

The amazing day continues with a track of 12 sessions by users, implementers, and developers of Apache Geode and GemFire.  There’s an optional break out in the morning for developers looking to get introductory hands-on training in how to get started with Apache Geode.

Speakers from companies such as Ampool, Capgemini, CIBC, Data Torrent, Murex, Societe Generale, Southwest Airlines, TEKSystems, zData will share their experience building solutions with Geode and GemFire.  Hear about use cases such as:

• real-time mobile apps
• streaming analytics
• fraud detection
• vehicle telematics
• financial risk computation
• adaptive logistics
• predictive analytics in manufacturing

Register today – space is limited!

We’ve already seen an outpouring of interest by experienced and new users of Apache Geode. We expect the summit to sell out.  Be sure to sign up today to reserve your spot in this amazing day.

Celebrating an Amazing 2015 for Pivotal Big Data Communities

In 2015, Pivotal released more than 6 million lines of code into open source, and we launched 4 major new open source projects:

• Apache Geode
• Apache HAWQ
• Apache MADlib
• Greenplum Database

This was a significant accomplishment by Pivotal in terms of engineering, legal, product development, and marketing efforts.

 

Since then we’ve seen quite a bit of interest in these nascent new communities in 2015.

 

2016 is our year to build them to critical mass of members with first official releases, simplifying the path to adoption and contribution, and driving awareness.

 

Here is Happy New Year message from Pivotal to the Pivotal Big Data Community.

Two Reasons Why In-Memory Data Grids Are A Must-Have For Apps at Scale

Cross posted from The Pivotal POV Blog…

Pivotal is proud to announce that Pivotal GemFire was cited as a leader in newly published the Forrester Wave™: In-Memory Data Grids, Q3 2015 report from Forrester Research. The report evaluated the respective in-memory solutions against 32 criteria in three high-level buckets of current offering, strategy, and market presence. GemFire was cited among the second-highest in the strategy category. Forrester provides the following comments about GemFire:

What to Expect from Pivotal at ApacheCon Europe 2015

Cross posted from The Pivotal POV Blog…

If anything exemplifies Pivotal’s all-in for open source direction, more than 20 sessions and events for ApacheCon Europe would be one such indicator. We are assembling a dozen open source contributors from 3 continents to help educate and inspire our fellow Apache contributors for projects we commit to such as: Apache Geode (incubating), Apache Ambari, Apache Hadoop, Apache Zeppelin (incubating), Apache Tomcat, Apache HTTP Server,Spring XD, Cloud Foundry and the Open Data Platform Initiative.

Students: “Be” The Job You Want Before You Graduate

“How to act like a baby unicorn”

It’s never been harder for a new grad to get a job in Silicon Valley, and it’s also never been easier. There are a number of forces that have converged to drastically change how new grads have to find their first job today versus how I was able to look for my first job years ago.

The Ambitious Apps at Amazing Scale Hackathon

In late June and through July 23, we held a virtual hackathon for Apache Geode with the help of the folks at DevPost.  Entitled: “The Ambitious Apps at Amazing Scale” hackathon, the goals were to:

1. Introduce new contributors to the Apache Geode community
2. Test a wide scope of new features and APIs being contributed to Apache Geode
3. Provide example applications to help inspire users how to use the Apache Geode technology

This video provides an overview of the contest:

Two eligible submissions were entered.

The winning submission, Dynamic Geode Warping was downright impressive.

Their video gives you a good overview of their work:

Bringing Commercial Software to Open Source

The following is a presentation I made with my colleague Cyrus Wadia to Pivotal’s Meetup group in June 2015 about what it took for us to spawn open source projects out of our long standing commercial products for big data.

And here’s the actual presentation: