In today’s web-enabled world, the use of Database Management Systems (DBMS) is extremely common and becoming even more necessary as each day passes. In most cases, the choice is your typical RDBMS such as MySQL, PostreSQL, SQLite or SQL Server. Some enterprise-level systems choose larger solutions such as Oracle.
As a company grows, it becomes a much bigger pain and seemingly less logical to maintain data in this manner. It requires more and bigger hardware, labor, decreases performance and becomes difficult to maintain and scale out. This can partly be blamed for the nature of relational systems.
Most RDBMS include the following high-level functions: querying, retrieving/writing data and replication. Each node acts on its own with identical or nearly identical copies of the data and, depending on whether the server is a slave, master or both, requires execution of the same query across the network in order to maintain the data. The query execution may come directly from the client or via replication from a master.
This is a relatively quick and easy start-up solution that requires less work as compared to a distributed system. The typical master-slave setup can, for the most part, be automated for new hardware. Just purchase similar hardware, increment the slave’s connection identifier, copy the data from another slave or snapshot and start it up.
To an extent, it’s also easy to scale. If you’re currently able to serve 2,000 queries per second, and your business grows to need 2,500, you simply add a new server. This is sufficient for many – probably most companies. But how about the need for 100,000, 300,000 or 2,000,000 queries per second? This is where the dwindling begins.
My rule of thumb is one master per four slaves at most – providing me with a 25% write-to-read ratio. Unfortunately, this also means I need some hefty, expensive hardware to do the job. A typical master for us at quepasa.com is dual, quad-core Xeons with 72GB of RAM. Since we are a heavy write-based system, we need fast disks; we also have a lot of data. In order to scale a solution like this, we have SANs in both data centers. Again, also very expensive. It works like a charm for now, but I’m constantly worried about what happens when we have 300,000 or more new users a day signing up. At what point does expense and maintenance start outweighing the benefit of this bulky hardware setup?
A few weeks ago I started research how DDMSs could work at our company. There were and still are many questions that I have to answer in this regard. What advantages and disadvantages will come with this new architecture? What hardware can I reuse efficiently with this new setup? What vendor do I choose to go with? What kind of code changes and culture shock will this introduce to the developers and DBAs?
Advantages
To begin with, this solution is extremely scalable. The general concept of a distributed database means expansion will be easier while not sacrificing reliability or availability. If I need to add more hardware, I should be able to pause a node or partition, copy it over and start the new hardware up (at a very high-level) without worrying about the rest of the system taking a massive beating and risking a crash or series of crashes.
Since this setup doesn’t require those massive machines we’re using right now, we can use commodity hardware that provides decent RAM (most of the expense of those bigger machines is here) and good CPUs. Off the top of my head, I imagine the hardware I’d consider for this would have four to eight GB of RAM and dual, quad-core Core i7s or possibly Xeons of the Westmere flavor. I definitely want to provide as many cores as I can to each node for higher throughput. This scenario likely means the bottleneck will now become the network itself. These nodes need to be able to communicate often, so a 10GbE network would eventually need to replace the 1 GbE. Speaking of which, I’ll move on to disadvantages.
Disadvantages
For our company, and probably all others in a similar scenario, the biggest disadvantage would be code and culture change. We’re used to writing code that connects to a database and executes a stored procedure that lives in the database and is written in SQL. Introducing this new architecture would completely change our environment. Stored procedures would likely be written in Java or another JIT language. The CRUD functionality would then execute that instead. This essentially means a rewrite of much of the site.
Building on that, we also introduce additional complexities into the architecture. This potentially means more cost in labor. Though the amount of time I put into weekend and late night/early morning maintenance renders that point moot.
We’ve also invested a lot of time and money into the hardware and architecture we have today. Introducing the concept of commodity hardware into the system could cause both economical and political backlashes. Fortunately, this likely appears more extreme than it really is. That bulky hardware could easily be sold at a decent price and the money from that used to purchase enough commodity hardware and then some. I’m not as worried about this aspect.
There are also solution-specific problems that are introduced. The concept of DDBMS itself is rather young and the lack of options and experience raises concern. This also means there’s a lack of standards to follow meaning moving to other solutions would be a major PIA at best. I would also be concerned with how the vendor treats concurrency control among the nodes. This is a major consideration in this type of environment.
While there are a few distributed solutions out there: Hadoop, Cassandra, Hypertable, Amazon SimpleDB, etc., one stands out in my opinion – VoltDB.
VoltDB is a next-generation SQL RDBMS with ACID for fast-scaling OLTP applications. It’s core is written in C++ and they provide many client connectors with Java the forefront. After looking into and reading more about them, I decided to query John Hugg (@johnhugg) for the potential of a PHP client. To my surprise, they already had one. John pointed me to the repository for the extension and I had a look at it. While John did an excellent job with the extension, it requires the use of a SWIG and it’s written in C++.
I asked John what he thought about a wrapper written in C and a pure C extension for PHP that is compiled against that library. It feels like a true “PHP solution” that could be added to PECL at some point and would eliminate the need for a SWIG. He was very excited about the potential for a more native solution.
That brings us to today. At this point, we’ve decided to research more into the potential for a C API. And once I get a better grasp on the core of VoltDB, I’ll be able to help John, et al. with a roadmap. The PHP extension will come after that.
I’m extremely excited about VoltDB and what opportunities it will bring to quepasa.com. I feel comfortable with the concept but skittish with the PHP integration. I’m confident that over the next year we’ll be able to come up with a good outcome and hopefully help others find their scaling solution in VoltDB.
If you’re interested in learning more about or helping with the C API and PHP extension efforts, visit the VoltDB community page and signup for the lists or leave a comment here. You can also download Volt and play with it in your own sandbox. It’s open source, so you can check out the core as well.
Update
After reading Alex Popescu’s post, I realized how bias my post sounded. For clarification purposes, I have not reached a final decision. In fact, I’ve chosen VoltDB as a starting point based on internal talents and familiarity. If VoltDB turns out to not be for my transition, I won’t use it. I do want the best technology but also have to keep things realistic.
This, however, does not stop the fact that I will be assisting VoltDB in their efforts with the C API and PHP extension. I like what they do and will continue to support them from an open source perspective. Your comments are welcome!
[...] This post was mentioned on Twitter by Will Fitch, Tim Callaghan. Tim Callaghan said: RT @willfitch: Anyone interested in distributed database systems? http://j.mp/fqkIvo [...]
I would like to point out that VoltDB requires everything to be resident in memory. So I don’t think that you are going to get away with your 8 GB of RAM per node. Just something to think about.
That would depend on the size of your partitions and the number of nodes in the cluster – (I was speculating, too)
Am I correct to think that if your company had used the Data Access Object Pattern for your database operations, then your migration to a DDMS would have been easier? Because you’d then only have to change your DAOs and not rewrite your whole site.
[...] Distributed Database Systems [...]
Did you take the lack of ORM support/capabilities of VoltDB into your considerations?
Thank you for this post. Very intresting.
@Pooria – Not really. Since database functionality is encompassed within stored procedures written in the application language, the ORM and other CRUD functionality is actually within it.
@Wonderer – It actually doesn’t take away ORM. Stored procedures are written in the language of your application code (Java, for example) and the ORM-specific functionality is maintained in those procedures.
[...] cases, the choice is your typical RDBMS such as MySQL, PostreSQL, SQLite or SQL Server. Some… [full post] Will Fitch Will Fitch's Blog databasesdistributed systemsmysqloperations 0 [...]
So is there any special reason you guys put your database functionality inside your databases? I hate it when people do that. Why not just use some ORM outside the database (like JPA or hibernate) to take care of most of the stuff?
Absolutely. We have a crew of expert application DBAs whose only job is to write and optimize SQL. This is a necessary step when fully optimizing an enterprise-level system. ORMs are great, but they aren’t tailored to your application and therefore are by their very nature “unoptimized”.