Sid Sidberry Reads

i prefer to handle data either in arrays or classes, or better yet, arrays with classes.

That said. get an xml to array script, like this one made by : Razzaque Rupom. It’s my favorite.

http://rupom.wordpress.com/category/phpclasses/

From: http://www.mysqlperformanceblog.com/2008/12/18/xtradb-benchmarks-15x-gain/

I guess it is first reaction on new storage engine – show me benefits. So there is benchmark I made on one our servers. It is Dell 2950 with 8CPU cores and RAID10 on 6 disks with BBU, and 32GB RAM on board with CentOS 5.2 as OS. This is quite typical server we recommend to run MySQL on. What is important I used Noop IO scheduler, instead of default CFQ. Disclaimer: Please note you may not get similar benefits on less powerful servers, as most important fixes in XtraDB are related to multi-core and multi-disks utilization. Also results may be different if load is CPU bound.

Check It Out:
XtraDB benchmarks – 1.5X gain in IO-bound load | MySQL Performance Blog

From: http://www.mysqlperformanceblog.com/2008/12/16/announcing-percona-xtradb-storage-engine-a-drop-in-replacement-for-standard-innodb/

Today we officially announce our new storage engine, “Percona XtraDB“, which is based on the InnoDB storage engine. It’s 100% backwards-compatible with standard InnoDB, so you can use it as a drop-in replacement in your current environment. It is designed to scale better on modern hardware, and includes a variety of other features useful in high performance environments.

Check It Out:
Announcing Percona XtraDB Storage Engine: a Drop-in Replacement for Standard InnoDB | MySQL Performance Blog

mysqlperformanceblog as a well written post discussing why mysql quality will never be what it once was. I honestly am more attracted to drizzle than mysql these days, but still a good read

I had a call with Monty the other day and I told him why I think MySQL Server Quality will never be the same again. I’ve been thinking a bit more about it and here is the extended list.

In particular I think MySQL Server will never be able to reach its original quality guidelines (see previous post) and even current release criterias will unlikely be ever reached with any sensible definition of what serious bugs are.

7 Reasons why MySQL Quality will never be the same | MySQL Performance Blog

So now I’m stuck wondering… which db do I use for new backend? I have a few months before its actually used, but I’ve seriously been wanting to try out mysql with a wafflegrid, but couchdb is just awesome…

Maybe I should just do both and hope I can toss two more servers into the mix before it actually launches….

everything waffle grid in one easy wiki
Main Page – The Waffle Grid Project

Now this is pretty wicked. They’ve used memcache as an L2 (secondary) cache for mysql. Wonder if there’d be any benefit if your running via vps?

Memcached as a L2 Cache for Innodb – The Waffle Grid Project
By matt

A few months ago I was at dinner with Yves Trudeau discussing what all consultants discuss in the late hours after a long day of hard work… how to improve performance and scalability. I brought up an idea to him to utilize memcached as an L2 cache for innodb. At first he was skeptical, but as we talked he was more and more intrigued by the idea. The idea was simple, add a set to memcached when something hit the LRU… then issue a get from memcached when you do not find the data locally stored in the buffer pool but before you read from disk. Starting from that point, you can work out any of the issues that would be sure to follow. So Yves continued emailing me asking me questions… then he sent me a note that he had made huge progress with the idea. Huge progress means that he wrote version 0.1 and had it working. That’s when the Idea really turned into a project.

We called it the Waffle Grid Project. Why? A waffle sort of looks like a grid diagram doesn’t it? And I like waffles, they happen to be very tasty. Having a working patch… We burned the midnight oil the last few weeks testing and fixing the code, building a proof of concept, and benchmarking it. So does it work? Yes, it does. Pretty well in fact. Take a look at some of the benchmarks below for a better idea.

Basically what this patch enables you to do is have a central node ( standard run of the mill database ) with several servers acting as remote L2 cache. An example: 1 Main Mysql server with 128GB of memory, 4 remote servers with 64GB of memory each… giving you a L2 cache of about 256GB. With a fast private network the L2 cache should return data faster then can be retrieved off of disk.

Here is what this would look like:

Memcached as a L2 Cache for Innodb – The Waffle Grid Project

When I first started this experiment I planned on using a custom php framework. Recently I realized that its kinda pointless to attempt to do something like this and at the same time lock myself into something that may not be the best solution….

enter google, research, testing, and little sleep. Result: codeigniter was the best choice. Huh? Here’s why.

snakes not on the plane

I should say from jump: If I knew and had the time to learn python / django it would have been the winner, sadly I do not.

ruby, rails, hype

I’m a fan of ror, and of course when I decided to take what I’ve created so far and migrate it into something usable I of course thought of ruby on rails. Sadly I saw more than a few hurdles.

First off ror doesn’t play we’ll with others, meaning you shouldn’t run rails in a shared environment. The whole concept behind this experiment is a small start-up with a shoe string budget and 2-3 websites / apps. Yes its possible to host multiple rails apps on one server / vps, but it’s not recommended. There is of course passenger, but that leads into the next point. php +1

Ror is a resource hog, there I said it. When compared to php, rails is more resource intensive than php. Of course the answer is to optimize and scale out, but remember I’m trying to keep the monthly hosting budget under 100-120 bucks (yes seriously shoe string). So php was the choice here.

So far php seems like the best choice for what I’m trying to do, but I needed more than just a few issues I could work around. You can’t work around speed and optimizing. ruby is faster than php via command line, but ror is not amazingly faster when used via web. When you add an opcode cache into the picture ror gets is butt kicked hands down. Of course this is comparing a language to a framework, enter codeigniter.

Codeigniter still out performs ror with an opcode cache. That is just one optimization and php shows drastic improvement.

I wasn’t ready to abandon ror yet, simply put: why use an imitator if you can have the real deal. Codeigniter is a great framework, but it makes more since to actually use rails right?

In the end the answer is no. Rails has a higher level of maintainability out the box, but does less code, easier maintenance, and of course the ror cool factor out weight slower speed, higher cost to deploy, and fewer production optimizations? No.

MVC while not as strict in codeigniter, is there. That combined with OOP will make using a php framework easier to maintain, not ror easy, but easy enough for production imho.

framework, shamework

So we now know why php won over ruby, but why pick codeigniter? I originally started out writing a custom framework, which is always fun to do from time to time to push yourself, but in the end you see the downside and benefits of doing so. Since this entire idea revolves around a small start-up we also need to take development time into consideration.

A custom framework takes alot of trial and error, coding, and recoding, and still more coding. Using a php framework I can reduce the time it takes to get up and running, while still building what I want and need. Yes there’s some overhead compared to a custom framework, but in the end a start-up isn’t yahoo. You should be back at the planning table long before you reach yahoo numbers. Thats not to say you shouldn’t plan for massive growth (hence scalable), just that traffic / users / data on yahoo’s scale is far beyond my scope of experience and even I know its another elephant to eat.

When comparing php frameworks, i was originally looking at cakephp, but after some research found it to slow and with to much overhead. There are other frameworks, but only codeigniter and cakephp met my needs. 2-1=1 (or less than one mr. V), so codeigniter was the winner.

One thing I should note is how poor most of the php frameworks I looked at perform out the box. Yes rails (and merb) are better out the box. Luckily a few simple optimizations speeds things up.

all your base are belong to theory

So far everything has been pretty much all theory, research, and testing. No worries, the meat and potatoes are coming shortly.

the next few posts will cover setting everything up(centos5): memcached / memcachedb, ngninx, apache, varnish, mysql, s3 + rsync, varnish -> s3 relations, etc.

If you feel like following along head over to http://linode.com and setup a few machines, or setup a team in vmware.

Worth Reading

Here are few articles that I came across while researching, some are more on topic than others, but all have some value.

PHP vs Java vs Ruby:
http://www.cmswire.com/cms/industry-news/php-vs-java-vs-ruby-000887.php

Ruby vs PHP performance (cli)
http://izumi.plan99.net/blog/index.php/2008/01/17/ruby-vs-php-performance/

Ruby on Rails Fans
http://shiflett.org/blog/2006/feb/ruby-on-rails-fans

The performance test of 6 leading frameworks
http://www.alrond.com/en/2007/jan/25/performance-test-of-6-leading-frameworks/

I haven’t had much time to post, or do anything fun in the past few months, but I’m currently working on a nginx, php, apache, varnish, memcachedb, mysql setup distributed between two servers. Why does this matter? Because I decided to upgrade my linode so I could slice up my resources into a nice little mini scalable network. Only problem is thats not how linodes work…

Linodes support team (Jim) had me upgraded 10 minutes after my upgrade request, then had me downgraded even quicker.

Yeah thats pretty good support. Better still is their control panel. I have yet to use their forums, and I’ve spoken to support once once before today.

If your looking for a good vps with intel processors, private network support, an awesome dns manager, and top support use these guys.

I still have my mediatemple [ http://mediatemple.net ] server, but for less than I’m paying I now have two servers with better stats and no bloat with linode [ http://linode.com ].

Check em out : http://linode.com

I’ve been seriously getting my kicks with scalability fora number of months, so why not start on ongoing series where I talk about what I’ve learned / found?

So welcome to to the first: Building Something Scalable – An ongoing experiment. This post covers caching. I’ll cover delivery in the next post.

Keep in mind language wise I’m using php, but the general advice should be sound, regardless of language. If you disagree with something or have a better way, feel free to comment.

So now that I’ve ranted off 4 topics, maybe I should expand on them a bit.

Use Caching

Caching is a good thing, but use caching is a pretty vague statement, so let’s expand.

Caching isn’t just a one ring to rule them all type of solution. Its actually a fixture of a number of different solutions, that work together to boost your site / applications overall performance.

Database caching

I consider database caching a 2 part solution. You have the mysql query cache, but I also like to have a server side query cache as well. Why? I tend to use oop and having a server side query cache allows me to cut some overhead both appilcation wise, and by preventing me from having to connect / query mysql.

The big issue with server side query caches is stale queries. The mysql query cache prevents stale caches automaticly, but with a server side query cache we’ll need to set a TTL (time to live). I tend to go with something really low like 5-10 seconds.

5-10 seconds may seem pointless, but it allows higher traffic applications to handle a number of requests with fewer queries to mysql. This takes some of the load from mysql, so  your database is performing under less load than it would have without the server side query cache.

There is plenty of information on the mysql query cache online, so fire up google and start researching. For your server side cache here a few things to keep in mind

1. keep your cache in a secure location. If your using a file cache this means outside of your web directory
2. hashing is a quick and painless way to uniquely id your queries. md5(’select * from table1′) will allows return md5(’select * from table1′) if done correctly.
3. prevent cache filename collisons.
4. do a light weight encoding on cache files. base64_encode / base64_decode are quick and easy to use. They’re not secure, but its a good idea to add some basic obfuscation
5. keep your TTL low. Your query cache should try to stay as fresh as possible.

Opcode Cache

Php is compiled / ran at runtime (when you request a page / script). Opcode caches store the compiled code so that your code doesn’t have to be compiled for every request. Opcode caches can increase your codes performence by up to 90%, but then again, any increase helps the overall perforence of your site / application.

There are a number of opcode caches avaible for php. I prefer xcache, but there are a number of other opcode caches available for php.

Static content cache

Static content unlike dynamic content is, well static. Your probably wondering: Why cache something thats already static? Simple, performence. Static content though is cached / served differently than dynamic content. I know this touching more on delivery, but its still worth mentioning.

Static content is often served through a CDN (content delivery network) or a web cache. A CDN and web cache act similarly, except that a cdn has a number of servers setup in various locations.

Content Delivery Network
A CDN acts just as it’s name says: It delivers your content via its network of servers. The CDN selects a server closest to the location of the user, and serves your content from that location. Whats the benifit? Faster delivery of your content. Is it worth it? That’s a question only you can answer. Do some research, compare the solutions, check you budget – and you’ll have your answer.

Web Cache
A web cache or reverse proxy simply put delivers your content faster. I’m not to well versed in the science of it all, but here’s a basic break down of what I do know:

Web caching software like vanish (in the past squid was the standard) handle servering static content better than apache, and with a smaller footprint. The web cache creates a cache of your content when requested and then delivers your content from its memory / disk cache.

The most obvious benefit from all this? Reduced server load. Apache is a resource hog (there I said it), but that will be covered in a future post in this series. By moving static content delivery to software created just for this task your freeing resources and of course getting content to users quicker.

Output Cache
So far we’ve looked at a number of ways to increase the speed of dynamic and static content, but there’s still one major item left out: Output Caching.

As your scripts / application generates pages, you can cache them to be served for future requests. Output caches in general can be as basic or complex as you need them. A few things to keep in mind.

1. stale content, your cache should have aTTL(time to live) that prevents it from serving stale content
2. filename collisons – your naming scheme should prevent filename collisons
3. store your cache outside of your web folder
4. logged in users / vs non logged in users – come up with a solution that deals with this.

Variable / Object Caches

Your code has objects and variables, often some of these objects are database intensive. An object / Variable cache is a way to store your objects and variables. The thing to keep in mind with these types of caches is speed.

It makes no sense to cache something like $var=1+1;. You can run that command quicker than you would access it from the cache. A good example of somethign to cache would a class object that runs a number of queries on the database, but accesses content that doesn’t change as often. By caching this object you can prevent a few database queries (or cache calls). Or a class object that generates a number of child class objects.

I could go on and on about this subject but lets get to the point. If your application is running on only one server use a file cache. If your application uses more than once server look into memcached / memcachedb.

Thats it.
Hopefully that was short and sweet, the next post will cover delivery.

Greg – Out