Scalable and distributed Storage CEPH

Wetopi Infrastructure Series Posts

Can you imagine an infinitely scalable storage?

It’s no longer a dream, it’s a reality!

Table of Contents

• Data storage is THE ISSUE of this century.
• Distributed storage, the solution.
• Why you should trust distributed storage?
• Our pick: Ceph 
  • Ceph distributed storage benefits
• Wetopi Ceph infrastructure
  • Your data is saved 3 times
• Leading the Ceph integration with Docker Swarm
• We didn’t find the perfect storage driver, so we created it!

Data storage is THE ISSUE of this century.

We realised that already back in 2015, when we moved Wetopi to a scalable infrastructure.

Since the beginning of the 21^st century people have started using technology in every aspect of life, adopting it more and more in order to make their lives easier. 

With this increased use, the demand to store and preserve the user data has also increased exponentially. 

 “Storage is the fundamental enabler of civilization. Writing (the first form of storage) enabled civilization. Digital storage enables digital civilization.”

Robin Harris from StorageMojo.

You use your phone to take pics, and to treasure these memories, upload them to web enabled storage like Google photos, for instance.

This past year Twitter announced that was expecting to transfer over 300 petabytes of data storage to the cloud. For comparison, at the end of 2017, the entire Internet Archive held almost 40 petabytes.

This example will help you to have an idea of the kind of data store these companies might be using.

With these figures you can imagine that data storage solutions needed to evolve to accommodate this ever-growing need for flexible and scalable storage resource.

Storage in single bare metal servers is fast and easy, you choose a fast SSD (solid-store drive) and voila! 

The problem with bare metal servers is that they need maintenance which also implies downtime and they are not easily scalable.

Distributed storage, the solution.

But what is distributed data storage?

Let’s check Wikipedia to give you a clear definition: 

“A distributed data storage is a computer network where information is stored on more than one node, often in a replicated fashion. It is usually specifically used to refer to either a distributed database where users store information on a number of nodes, or a computer network in which users store information on a number of peer network nodes.”

Wikipedia

Imagine a bunch of bare metal servers on a network (named cluster), with virtualization software that lets you easily create Virtual Machines. 

Virtualization software allows you to run two or more operating systems using only one PC. So it’s like two (or more) computers in one. 

Distributed storage lets you attach your data to a virtual server, and the best thing is that you can change your virtual server in size, placement, operating system, etc. 

In a nutshell, distributed data storage means that your data volume can be instantly moved from one location to another. 

Why you should trust distributed storage?

In the past, portable, fast, and scalable storage was usually provided with a traditional SAN (Storage Area Network). These huge storage systems were expensive and difficult to manage and upgrade. 

Scaling and upgrading was difficult, and the architecture was susceptible to considerable vendor lock-in or parts obsolescence.

In the past 15 years since the distributed storage alternative appeared, it has steadily grown in popularity, performance, stability, scalability, and features.

Distributed storage usually implies redundancy, and this gives us the advantage of reading from multiple sources at the same time.

The idea behind “distributed” is to give you the advantage of using a multitude of standard servers, which behave as one huge virtual storage system. 

Our pick: Ceph

We have intensively tested and analysed different distributed storage solutions.

And we came to the conclusion that Ceph is an incredible beast!  After some months of testing we were really impressed!

Ceph is THE leading open-source software-defined storage solution on the market, with countless supporters from research data centres of every size and industry. 

Some of the reasons Ceph is such an ideal solution are that it’s flexible, inexpensive, fault-tolerant, hardware neutral, and infinitely scalable which makes it an excellent choice for organisations of any size.

Ceph’s algorithm, CRUSH, liberates storage clusters from the scalability and performance limitations imposed by centralised data devices. 

CRUSH replicates and rebalances data within the cluster dynamically—eliminating this tedious task for administrators, while delivering high-performance and infinite scalability.

Ceph is the result of hundreds of contributors and organisations working together in the best practices of Open Source. Some of these organisations are Intel, Fujitsu, Sandisk, just to name but a few.

Ceph distributed storage benefits

As we mentioned before Ceph is flexible, inexpensive, fault-tolerant, hardware neutral, and infinitely scalable. On top of all this Ceph provides us with:

• Support for multiple storage types: object, block, and file systems. Regardless of the type of work being carried out, the resulting files, blocks and/or objects can all live in harmony in Ceph.
• Natively supporting hybrid cloud environments.
• Open-source software – Freedom from vendor lock-in and product obsolescence
• Fault tolerance: there’s no need to buy redundant hardware in the event a component fails. Ceph’s self-healing functionality quickly replicates the failed node, ensuring high availability and data redundancy. 

The only down side of Ceph is that it’s extraordinarily complicated to run correctly. In other words, to take advantage of such an amazing tool you need a deep understanding of Ceph. 

We’ll try to sum up as clearly as possible how we work with Ceph.

Wetopi Ceph infrastructure

At Wetopi all Virtual Machine Clusters consume storage simultaneously from 2 pools. 

1. SATA (Serial Advanced Technology Attachment storage: storage based on multiple SATA disks), the “non-performant” pool.
   This pool of storage is consumed in infrastructure servers with tasks that don’t require massive performance.
   
2. The SSD “the performant”. Based on an aggregate of Solid Storage Disks. This pool provides storage to all virtual machines used by the WordPress end user and all infrastructure virtual machines requiring massive performance.

Your data is saved 3 times

Both pools are configured to store the data in replica 3. This way, in the virtually impossible event that one or even two servers storing would break some of your WordPress objects, nothing would stop or get lost.

With 3 copies of your objects your work is secure!

Let’s see what happens with your data when you save something on WordPress working with Ceph:

1. The data goes to the SSD pool and the CRUSH algorithm breaks it assigning different pieces of data to different Placement Groups.
2. The Placement Group uses the CRUSH algorithm to send your data to the primary OSD (Object Storage Daemon).
3. The Primary OSD replicates the Placement Group to the second and third OSD
4. When your server receives the “write” confirmation from Primary and Secondary ODS, then your data is considered “safe”.

Leading the Ceph integration with Docker Swarm

Your WordPress servers at Wetopi are containerized using Docker.

Docker Swarm is a cluster orchestrator. It’s in charge of maintaining your WordPress server alive. If your server stops due to a problem: 

1. The orchestrator will try to rescue your service.
2. If service fails due to a problem on its current placement, it looks for a new physical placement, moving your server to a new functional node.

To let the Docker orchestrate and freely move your server around the cluster, Docker has to be capable of managing your storage volumes.

To do this efficiently, we developed our own Docker storage Driver.

We didn’t find the perfect storage driver, so we created it!

Do you want to know why?

1. Docker is a platform in constant evolution. We wanted a storage plugin capable or running with the latest docker releases.
2. The existing RBD volume drivers (RADOS Block Device) were old and had lots of other software dependencies. Our goal was to have a lightweight and fast driver 

Here you have the link to download our Docker Swarm RBD plugin https://github.com/wetopi/docker-volume-rbd

This driver is under MIT License so you can use it for free or contribute!

We hope that after having read this post you have a better idea about how distributed storage work and why is THE solution to the storage issue.

Let us know in the comments if you still have questions about it. We’ll be happy to help!

Don’t you have an account on Wetopi?

Free full performance servers for your development and test.
No credit card required.