Two Reasons Why In-Memory Data Grids Are A Must-Have For Apps at Scale

Cross posted from The Pivotal POV Blog…


Pivotal is proud to announce that Pivotal GemFire was cited as a leader in newly published the Forrester Wave™: In-Memory Data Grids, Q3 2015 report from Forrester Research. The report evaluated the respective in-memory solutions against 32 criteria in three high-level buckets of current offering, strategy, and market presence. GemFire was cited among the second-highest in the strategy category. Forrester provides the following comments about GemFire:

Pivotal Software open sources GemFire to spur community innovation. GemFire has it all—data and compute features that have been tested under the most challenging conditions. GemFire was one of the original IMDG that was quickly adopted by financial customers due to its compute and data capability. Pivotal Software has contributed GemFire to the Apache open source community as Geode.

While we believe that this is a fine compliment unto itself, what struck me about the report is this advice from Forrester:

Build IMDG’s into Your Application From the Outset—AD&D pros should not make the mistake of turning to IMDGs only when performance at scale becomes an issue. It will become an issue sooner or later.

This is an insight that we feel many of GemFire’s customers have already learned. In many cases, once a customer puts their first large scale GemFire-based application into production, they typically deploy a GemFire cluster as a standard component of their custom application infrastructure.

There are a number of reasons to do this. Sometimes all a customer wants is to ensure extremely fast performance, which GemFire’s in-memory storage provides as a cache for queries or computation. Other times they need more sophisticated capabilities, such as support for committing ACID transactions at high concurrency, low-latency continuous queries, or the ability to scale an application globally supported by multiple replicated clusters.

This is why Pivotal recently released a GemFire service for Pivotal Cloud Foundry, so that Cloud Foundry developers can easily add this component to their stack when they need to scale their applications to handle high concurrency.

Why Pivotal Believes In-Memory Data Grids Are Must-Have Components for Apps at Scale

It is our view that Forrester lays out twin drivers for why in-memory data grids should be a typical architectural component for many application stacks:

Modern App experiences require more and faster data access” Forrester’s observes “the problem is that an app’s read/write access will never be faster than the slowest data source.” Thus developers would need a caching layer that bridges across multiple data sources of varying access speeds.

Our experience with GemFire customers also suggests that in-memory processing also allows app developers to provide smarter, context-aware applications. For example, ranked or dynamic lists can reflect in-the-moment states, or arranged predictively according to what the user is likely to be trying to achieve with the app.

Scale and concurrency stress architectural components” Forrester writes about the ever increasing number of mobile phone users as one example of high concurrency that challenges yesterday’s scale-up database architectures.

We call this the need for “compulsive” scale. This phenomenon will only continue to grow as Internet of Things applications connect many more devices. As these IoT technologies come online, the need for concurrent automated processing at very low latency will increase exponentially.

Forrester also does a great job of explaining the common elements of IMDGs that help applications scale to solve these drivers:

  • “Use RAM to reduce latency of data read/write access”
  • “Distribute data across nodes to scale linearly”
  • “Replicate data across nodes to achieve high-availability”
  • “Synchronize data across data centers for disaster recover”

Forrester lays out a number of use cases suitable for IMDG’s. Pivotal agrees with these, but sees opportunity for GemFire in four major types of applications:

  1. Extreme transaction processing where time sensitive SLAs require elastic capacity for committing and accepting or denying applications in any volume with a second or less latency. These tend to be mission critical processes, so disaster recovery and failover capabilities available through cluster to cluster replication over WAN are very important. This capability also allows the ability to scale globally with multiple clusters processing transactions from different localities, or handling different parts of the application workload.
  2. Big Data Operationalization, which can be argued as a specific case of “data services fabric for real-time data integration”. However the requirement we see here isn’t just providing a cache for an integrated dashboard. We see inbound integration and writing as key to for writing historical data for data warehousing or to HDFS. This is one reason we created the GemFire to Greenplum Databaseintegration. Stream processing applications and IoT would also fit into this category. Features such as continuous query allow customers to define low-latency, custom event triggers to create alerts or fire off automated handling functions.
  3. Resilient Web App Acceleration, which specifically leverages the session state caching capabilities of GemFire, and the ability to span this between data centers. You can use this capability natively, or within a Cloud Foundry-based application.
  4. Scalable Persistence and High Performance Transactions for Cloud Native Applications, which I would classify as a latter-day version of their use cases of “primary store for modern apps” and “NoSQL database at in-memory speed”. However, this NoSQL primary store provides ACID transactions, and runs inside Cloud Foundry.  How this Cloud Native integrated offering will be the building block for future microservice-based in-memory processing is maybe the most exciting part of this story.

There’s a lot of good detail in this Forrester Wave we have not been able to cover, taking an in-depth look at in-memory data grids and the various technologies that they can drive. Download it today and read it for yourself!

Learn More:

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: