mySQL, noSQL, and Key Value datastores
Monolithic RDBMs are losing ground to key-value data stores, particularly persistent distributed in nature. mySQL mounting problems was perhaps the key reason (pun intended) people looked elsewhere. Google's brilliant engineers realized that a key/value data model can satisfy the needs of almost every class of application that needs a datastore backend.
Key/value datastores are simple to build, easy to understand, easy to optimize, easy to scale. The, now famous, CAP theorem states that it is not practically possible to guarantee consistency, availability and partitioning resilience/tolerance all at the same time; one of those traits has to be sacrificed. Again, most applications really do not require all three to function. The CAP theorem is most likely derived from the Project Triangle mode.
Most web-based applications are built on simple data models. Most web-applications eventually suffer from service capacity and availability issues(i.e scalability woes). It is trivial to scale out(vertically) application logic processing(application servers), HTTP requests processing(web servers, load balancers).
It is not easy to scale out an RDBMS. Some expensive systems(Oracle, etc) provide ways to address those issues (e.g Oracle RAC) but its expensive to deploy them, and most of them rely on a shared everything setup which just doesn't work in the long run. (Shared nothing is really the way to go).
Google released a bunch of papers ( actually, a bazillion of papers ), many of them defining and shaping the development of future related technologies. Namely, the papers describing GFS, BigTable, MapReduce (and of course, the paper the changed everything, "The Anatomy of a Large-Scale Hypertextual Web Search Engine" ) steered everyone to the right direction.
In the datastores domain, Hadoop/HBase, Radix, Cassandra and others, based on BigTable and Amazon's Dynamo papers, all relying on the simple key/value datastore model, are gaining market share - rightly so. Coupled with Memmache and similar services(in-memory key/value stores) they are solving the problems of service capacity and availability. This is a paradigm shift. Its a downhill for heavy-footprint, complex and inflexible datastore systems. They wont go away but will not be such a valuable(pun intended) component in tomorrow's technology landscape.
We are going to gradually migrate from RDBMs - though, we are not relying that much on them nowadays - to a key/value datastore (we are currently building one, also based on BigTable and Dynamo ). If nothing else, those simple systems are both simple and beautiful (for the most part).
Saturday, 13 March 2010 8:14 pm
New Theme, iPhone, Google AppEngine
My brother provided me with a theme for my blog. Its pretty clean and simple - yet not a simple or clean as I would have wanted it to be, but that's entirely my fault. Its a matter of modifying the structure of the various elements and using font families and colors that make sense.
I purchased two iPhones from Las Vegas ( Thank you for the invitation Patrick ). I used to dislike cell phones with a passion. Especially those engineered by Nokia. Complicated for no reason, cumbersome to use, fancy for the sake of being fancy and loaded with a gazillion crappy applications and 'services'. The only cell phone I actually liked was the original Nokia phone ( short-lived moment of glory for them ) used in the Matrix 1 movie. So, naturally, my expectations were rather low when it came to putting the iPhone to the test.
"The iPhone is a revolutionary mobile phone". It actually is. Everything just works, supported by an ultra sleek UI, robust facilities and solid design decisions. It is by far the best mobile device I ever used, far surpassing any expectations I may have had.
Amazon kick-started the cloud computing era by introducing an ever expanding array of facilities and services, from S3 to EC2, to SimpleDB. Microsoft is entering the game with SSDS. Google made available a dozen APIs and WebService as a means to interfacing with their core services but everyone knew Google would come after Amazon and co, big time. It did. What is perhaps the most important benefit and side-effect of the availability of such a platform is that the everyone can build any web application without having to shelling out for the kind of resources that would have made this application possible. The AppEngine service is going to provide everyone with free access to resources and documentation - all one would need to do is signup with them, build the application on his computer using the provided SDK and then push it back to the cloud. Once the application gets successful (say, 4-5 million page views / month ) that said developer would pay Google for access to more resources. Everyone wins.
I am looking forward to similar offerings from IBM and Sun. For those who are into buzzwords, Web3.0 is here.
Tuesday, 8 April 2008 10:08 pm
Enterprise level mashups
We are seriously considering offloading some of our data storage and hosting needs to Amazon Web Services, and if that works out - it most certainly will - we 'll offload some more and take it from there.
Amazon has quietly been building a google-grade, highly scalable, reliable and distributed in nature infrastructure, while most folks would disregarding it as an e-tailer. The largest e-tailer, but 'just' an e-tailer nevertheless. They were wrong. Bezos is not to be taken lightly.
Secure Storage Service(S3) is the AWS's flagship service. In short, it allows for 'unlimited' data storage and access of arbitrary data objects ( AKA files ) consolidated into buckets ( psudo-directories ). Its trivial to access the service via its XML (REST and SOAP ) based API. Its super-cheap ( especially if you happen to be dealing with the kind of price tags ISPs in Greece attach to their related offerings ) and, how about that, it works.
It's been suggested before and it gets more obvious by the day. With a very little sum of money, lots of will and dedication and, preferably, with a good idea to ago along, a bunch of guys can build the next Youtube. Delegate the storage and hosting to AWS, setup a few servers ( you can 'hire' virtual servers on AWS's Elastic Compute Cloud for 10 cents / hour ) to do your bidding, use blogs for free ( and highly effective ) PR, and hey-presto, money and fame are pouring in.
Google, Yahoo, Microsoft and an ever increasing companies are offering similar services ( software as a service, where did I here that before ) which you can bring in together, mash-them-up and you are set. Scalability, distribution and reliability is implicitly guaranteed by the service providers. We are entering this business sooner or later. All our services ( we love modules and components here ) have been designed to be accessible via XML (XML-RPC, REST, SOAP, JSON etc) . Also, we are big on clusters and fault tolerant systems so just about everything is already in place. In fact, everything has been in place for over 5 or so years, since we switched to the new architecture model.
We live in interesting times. Interesting things happen all the time. Go ahead, build the next big thing and contribute to that interest factor. Trust me, you are going to have a whole lot of fun doing it.
Saturday, 16 June 2007 6:52 pm