SCM DIMM wars - article by StorageSearch.com
Here's some of what Diablo said in its press release:-
"The revolutionary technology packs 4x the capacity of the largest DRAM modules, delivering greater capability on fewer servers and lowering datacenter costs by up to 70%.
"The same system memory slots that now hold 128 or 384 gigabytes of DRAM memory can house up to 4TB of Memory1 and process data-intensive applications that were previously beyond reach. These efficiencies are driven by Diablos breakthrough in memory design, which replaces expensive DRAM with low cost, high capacity flash, while delivering the performance an enterprise demands.
"Memory1 is deployed into standard DDR4 DIMM slots and is compatible with standard motherboards, servers, operating systems and applications. Memory1 is shipping now to select customers and will be broadly available this fall."
"Memory1 brings the low cost and high capacity of flash to large-scale enterprise and datacenter customers. It is ideal for environments that require large memory footprints per server for workloads, such as big data analytics and complex Web applications. The average Memory1 use case enables a 4 to 1 server reduction, and one customer use case requires 90% fewer servers. Diablo will initially focus on delivering Memory1 to cloud and hyperscale datacenters, which stand to see significant economic benefits because of their scale."
But as I said to Diablo - when you've got a technology story like this - likely to have a big impact on SSD history - a few days or weeks delay in the retelling of the story don't really matter.
My gut feel assessment is this.
Diablo's Memory1 is the most significant development in the enterprise flash market in the past 3 years (since the widespread impact of adaptive R/W and DSP controller technology) and Diablo's Memory1 is the most significant development in the enterprise application server market in the past 8 years (since the migration of most enterprise SSD installed capacity to flash away from RAM and - in particular - the launch of the first flash based PCIe SSDs by Fusion-io in September 2007).
But let me clarify something important.
Memory1 is not an SSD.
Memory1 is a volatile memory which replaces DRAM with a flash array.
You'll still need some real DRAM in your system. Diablo currently recommends a ratio of no more than 10 to 1 (Memory1 to RAM). And there are OS limitations too. Currently support is limited to Linux although there are plans to support other major platforms too.
the big question - how does flash replace RAM?
Diablo didn't want to say much about how they use the flash.
Diablo told me - "We are not disclosing the flash technology, only that we are using widely available NAND flash. We employ several technologies to ensure endurance, an example is that we deliver volatile storage, which is less taxing on the media than persistent storage."
Diablo declined to reveal how much overprovisioning they use inside the flash array. But they did answer my question about warranty - 3 years.
A year later I learned the answers and published them here - SSD aspects of Diablo's Memory1 and DMX - which explains how their DRAM controller and flash controller can reliably replace DRAM with a conventionally rated DWPD flash. ...read the article
This bring us back to the question.
what is Enterprise DRAM?
At a top level - DRAM - seen from the applications server point of view - has complex and changing latency characteristics.
This is because traditionally it is the sum effect of a bunch of different latencies (as experienced by the software) due to a hierarchy of latencies which include:-
- memory caches on the CPU chip,
- caches external to the CPU chips,
- real DRAM (the kind of modules you see plugged into the motherboard) and
- emulated DRAM (the virtual memory which gets swapped in and out of HDDs and SSDs).
In the past year or so in articles which I have collected in are you ready to rethink enterprise DRAM architecture? - it's been clear that researchers have been evaluating the memory and server experience and have realized that - depending on the application software - it's possible to get similar performance effects to traditional DRAM and CPU sets in other ways.
The 2 challenges for the DRAM market come from 2 factors:-
- DRAM latency at the hardware level (as seen by local CPUs) is worse than it
was in the year 2000.
There are many reasons for this - the chief being that enterprise memory systems have been optimized for density and cost rather than speed. And the growing complexity of multi processor cache designs means that worse case latencies are much longer than you might think. See more here - latency loving reasons for fading out DRAM in the virtual memory slider mix.
- DRAM latency - as seen by applications - is really the "virtual
memory" latency which arises from the cumulative impact of several
different system behaviors. These include upstream storage, cache properties
and the software stacks. In effect all legacy applications have been "trained"
during the past decade or so to operate with a DRAM latency which has
statistical latency properties which are good enough rather than guaranteed.
During the first 5 years or so of the PCIe SSD market it became clear that making storage latency faster (than the HDD models assumed in rotating storage architecture) could enable trade offs in memory size in which DRAM could be replaced by flash within certain operating bands and achieve similar application performance at much lower operating costs.
What we've got here is a new memory type - which will not only take business away from traditional DRAM and CPUs (as a special case extrapolation of the CPU-SSD equivalency reasons which I told you about in 2003) but Memory1 will also take business away from that 3D PCM memory which Intel and Micron have been talking about too - for the simple reason that flash is a cheap and more mature technology.
This new technology from Diablo is more significant than what we had been led to expect from their previews a year ago.
It doesn't replace memory channel SSDs or hybrid DIMMs - it's a new product type - flash as RAM - which could lead to a market as significant in revenue as the whole PCIe SSD market.
I'm guessing then that Diablo will stay the #1 most researched company in the SSD ecosystem for a while as a result.
Competitors to Diablo's Memory1?
As I said above - many other SSD companies have been eyeing exactly the same applications problems which Diablo's Memory1 is designed to satisfy:- lowering the cost of server populations which rely on access to extreme amounts of low latency RAM in order to fulfill their business purposes.
(...Later:- You can see a growing double digit list of competitors in the sidebar article on the right.)
While there are currently no exact form factor matches to Memory1 which give similar memory density - there are other ways to get similar results. And if you regard the server box as the socket footprint (which it really is - if you're planning to buy 10K or 100K servers) then DIMM socket compatibility - while convenient - is not essential.
My preliminary list of potential hot competitors in the Diablo Memory1 market space goes like this.
Flash based solutions.
In the current state of the memory market - nand flash is the cost leader. Alternative approaches include:-
- software which remaps big RAM memory into flash SSDs.
For example:- SanDisk's ZetaScale. This operates with a wide range of SSD interfaces - so users have a lot of freedom to optimize their latency / cost profile. At the extreme limits Memory1 may be faster in latency terms than most PCIe SSDs - but my guess is that some kind of future ioDrive like product (possibly optimized to strip away non volatility) would give very similar performance.
Other SSD software companies which until recently have been in stealth mode are also coming into the latency lowering flash market.
- in-situ
processing - combined with PCIe SSD platform.
Companies which have been talking about this since last year include:- NxGn Data, Memblaze and others. And earlier this year further validation of the concept came from researchers at MIT who showed that flash can come close to RAM rich server performance by using clunky app-specific FPGAs combined with flash.
- Netlist? - This company which was embroiled for over a year in legal actions aimed at Diablo's first product - the MCS SSDs - is one of several in the hybrid DIMM market which might make a future pitch in this market. Having said that there's a world of difference between these DIMM compatible RAM SSDs and Memory1 - where the RAM which is used is actually in another socket supplied by someone else. Other companies in this category include Micron. But due to the market risks of shooting its other products in the foot I think Micron will prefer to position its response to this application in different ways first (with higher priced solutions).
These will probably cost more than the flash based competitors.
- Already mentioned above - high density 3D PCM from Intel and Micron. Details at the present time are sketchy. But it's reasonable to assume this will initially be offered at much higher cost than flash solutions - because it's a new technology and unlike Memory1 - a key characteristic it offers is non volatility.
Another contender which converges with all the technologies above is PMC whose NVM Express over RDMA includes elements of PCIe fabric but interoperate the company's low density RAM SSD and flash SSD products.
Together these can be combined to build low cost low latency fabrics for small server clusters at a starting price point probably below PLX and much lower (scale) than A3CUBE.
Currently these fabric solutions enable users to extend the DRAM space at low microsecond latencies outside the rack rather enabling more RAM in the same rack. But I think they could also be viable technology launchpads for future PCIe connected flash memory similar to Diablo's Memory1.