Sid Sidberry Reads

It’s the holidays and everyone is all about giving, why not give to the honeypot project?

we all have a few domains we’ll never use for mail, why not put them to use killing spammers? Well not the death kinda killing, but you get the point…

From: http://www.projecthoneypot.org/manage_mx_entries.php#donate

In order for us to create honey pot addresses that spammers cannot tell from real addresses we need as many donated domains as possible. You can help us by donating an MX entry for a domain you control. You may setup your donation such that it will not affect existing mail or web traffic coming to your site. For more information, please see our FAQ.

Check It Out:
Project Honey Pot

When I first started this experiment I planned on using a custom php framework. Recently I realized that its kinda pointless to attempt to do something like this and at the same time lock myself into something that may not be the best solution….

enter google, research, testing, and little sleep. Result: codeigniter was the best choice. Huh? Here’s why.

snakes not on the plane

I should say from jump: If I knew and had the time to learn python / django it would have been the winner, sadly I do not.

ruby, rails, hype

I’m a fan of ror, and of course when I decided to take what I’ve created so far and migrate it into something usable I of course thought of ruby on rails. Sadly I saw more than a few hurdles.

First off ror doesn’t play we’ll with others, meaning you shouldn’t run rails in a shared environment. The whole concept behind this experiment is a small start-up with a shoe string budget and 2-3 websites / apps. Yes its possible to host multiple rails apps on one server / vps, but it’s not recommended. There is of course passenger, but that leads into the next point. php +1

Ror is a resource hog, there I said it. When compared to php, rails is more resource intensive than php. Of course the answer is to optimize and scale out, but remember I’m trying to keep the monthly hosting budget under 100-120 bucks (yes seriously shoe string). So php was the choice here.

So far php seems like the best choice for what I’m trying to do, but I needed more than just a few issues I could work around. You can’t work around speed and optimizing. ruby is faster than php via command line, but ror is not amazingly faster when used via web. When you add an opcode cache into the picture ror gets is butt kicked hands down. Of course this is comparing a language to a framework, enter codeigniter.

Codeigniter still out performs ror with an opcode cache. That is just one optimization and php shows drastic improvement.

I wasn’t ready to abandon ror yet, simply put: why use an imitator if you can have the real deal. Codeigniter is a great framework, but it makes more since to actually use rails right?

In the end the answer is no. Rails has a higher level of maintainability out the box, but does less code, easier maintenance, and of course the ror cool factor out weight slower speed, higher cost to deploy, and fewer production optimizations? No.

MVC while not as strict in codeigniter, is there. That combined with OOP will make using a php framework easier to maintain, not ror easy, but easy enough for production imho.

framework, shamework

So we now know why php won over ruby, but why pick codeigniter? I originally started out writing a custom framework, which is always fun to do from time to time to push yourself, but in the end you see the downside and benefits of doing so. Since this entire idea revolves around a small start-up we also need to take development time into consideration.

A custom framework takes alot of trial and error, coding, and recoding, and still more coding. Using a php framework I can reduce the time it takes to get up and running, while still building what I want and need. Yes there’s some overhead compared to a custom framework, but in the end a start-up isn’t yahoo. You should be back at the planning table long before you reach yahoo numbers. Thats not to say you shouldn’t plan for massive growth (hence scalable), just that traffic / users / data on yahoo’s scale is far beyond my scope of experience and even I know its another elephant to eat.

When comparing php frameworks, i was originally looking at cakephp, but after some research found it to slow and with to much overhead. There are other frameworks, but only codeigniter and cakephp met my needs. 2-1=1 (or less than one mr. V), so codeigniter was the winner.

One thing I should note is how poor most of the php frameworks I looked at perform out the box. Yes rails (and merb) are better out the box. Luckily a few simple optimizations speeds things up.

all your base are belong to theory

So far everything has been pretty much all theory, research, and testing. No worries, the meat and potatoes are coming shortly.

the next few posts will cover setting everything up(centos5): memcached / memcachedb, ngninx, apache, varnish, mysql, s3 + rsync, varnish -> s3 relations, etc.

If you feel like following along head over to http://linode.com and setup a few machines, or setup a team in vmware.

Worth Reading

Here are few articles that I came across while researching, some are more on topic than others, but all have some value.

PHP vs Java vs Ruby:
http://www.cmswire.com/cms/industry-news/php-vs-java-vs-ruby-000887.php

Ruby vs PHP performance (cli)
http://izumi.plan99.net/blog/index.php/2008/01/17/ruby-vs-php-performance/

Ruby on Rails Fans
http://shiflett.org/blog/2006/feb/ruby-on-rails-fans

The performance test of 6 leading frameworks
http://www.alrond.com/en/2007/jan/25/performance-test-of-6-leading-frameworks/

I’ve been seriously getting my kicks with scalability fora number of months, so why not start on ongoing series where I talk about what I’ve learned / found?

So welcome to to the first: Building Something Scalable – An ongoing experiment. This post covers caching. I’ll cover delivery in the next post.

Keep in mind language wise I’m using php, but the general advice should be sound, regardless of language. If you disagree with something or have a better way, feel free to comment.

So now that I’ve ranted off 4 topics, maybe I should expand on them a bit.

Use Caching

Caching is a good thing, but use caching is a pretty vague statement, so let’s expand.

Caching isn’t just a one ring to rule them all type of solution. Its actually a fixture of a number of different solutions, that work together to boost your site / applications overall performance.

Database caching

I consider database caching a 2 part solution. You have the mysql query cache, but I also like to have a server side query cache as well. Why? I tend to use oop and having a server side query cache allows me to cut some overhead both appilcation wise, and by preventing me from having to connect / query mysql.

The big issue with server side query caches is stale queries. The mysql query cache prevents stale caches automaticly, but with a server side query cache we’ll need to set a TTL (time to live). I tend to go with something really low like 5-10 seconds.

5-10 seconds may seem pointless, but it allows higher traffic applications to handle a number of requests with fewer queries to mysql. This takes some of the load from mysql, so  your database is performing under less load than it would have without the server side query cache.

There is plenty of information on the mysql query cache online, so fire up google and start researching. For your server side cache here a few things to keep in mind

1. keep your cache in a secure location. If your using a file cache this means outside of your web directory
2. hashing is a quick and painless way to uniquely id your queries. md5(’select * from table1′) will allows return md5(’select * from table1′) if done correctly.
3. prevent cache filename collisons.
4. do a light weight encoding on cache files. base64_encode / base64_decode are quick and easy to use. They’re not secure, but its a good idea to add some basic obfuscation
5. keep your TTL low. Your query cache should try to stay as fresh as possible.

Opcode Cache

Php is compiled / ran at runtime (when you request a page / script). Opcode caches store the compiled code so that your code doesn’t have to be compiled for every request. Opcode caches can increase your codes performence by up to 90%, but then again, any increase helps the overall perforence of your site / application.

There are a number of opcode caches avaible for php. I prefer xcache, but there are a number of other opcode caches available for php.

Static content cache

Static content unlike dynamic content is, well static. Your probably wondering: Why cache something thats already static? Simple, performence. Static content though is cached / served differently than dynamic content. I know this touching more on delivery, but its still worth mentioning.

Static content is often served through a CDN (content delivery network) or a web cache. A CDN and web cache act similarly, except that a cdn has a number of servers setup in various locations.

Content Delivery Network
A CDN acts just as it’s name says: It delivers your content via its network of servers. The CDN selects a server closest to the location of the user, and serves your content from that location. Whats the benifit? Faster delivery of your content. Is it worth it? That’s a question only you can answer. Do some research, compare the solutions, check you budget – and you’ll have your answer.

Web Cache
A web cache or reverse proxy simply put delivers your content faster. I’m not to well versed in the science of it all, but here’s a basic break down of what I do know:

Web caching software like vanish (in the past squid was the standard) handle servering static content better than apache, and with a smaller footprint. The web cache creates a cache of your content when requested and then delivers your content from its memory / disk cache.

The most obvious benefit from all this? Reduced server load. Apache is a resource hog (there I said it), but that will be covered in a future post in this series. By moving static content delivery to software created just for this task your freeing resources and of course getting content to users quicker.

Output Cache
So far we’ve looked at a number of ways to increase the speed of dynamic and static content, but there’s still one major item left out: Output Caching.

As your scripts / application generates pages, you can cache them to be served for future requests. Output caches in general can be as basic or complex as you need them. A few things to keep in mind.

1. stale content, your cache should have aTTL(time to live) that prevents it from serving stale content
2. filename collisons – your naming scheme should prevent filename collisons
3. store your cache outside of your web folder
4. logged in users / vs non logged in users – come up with a solution that deals with this.

Variable / Object Caches

Your code has objects and variables, often some of these objects are database intensive. An object / Variable cache is a way to store your objects and variables. The thing to keep in mind with these types of caches is speed.

It makes no sense to cache something like $var=1+1;. You can run that command quicker than you would access it from the cache. A good example of somethign to cache would a class object that runs a number of queries on the database, but accesses content that doesn’t change as often. By caching this object you can prevent a few database queries (or cache calls). Or a class object that generates a number of child class objects.

I could go on and on about this subject but lets get to the point. If your application is running on only one server use a file cache. If your application uses more than once server look into memcached / memcachedb.

Thats it.
Hopefully that was short and sweet, the next post will cover delivery.

Greg – Out

Yes, I know there are much more great FireFox plugins but I am sharing only those I am using myself and find really useful.

Before I start, here is an overview of what I am going to talk about:
SEOQuake Any page instant SEO analysis
SearchStatus Google, Alexa, Compete Rank Toolbar
Google toolbar Google search and suggestions
Rank Checker button Instantly checks a site SE rankings
User Agent Switcher Change the user agent
SEO for FireFox Exhaustive SEO analysis within search results
Compete.com Site traffic information
Web Developer Instant access to image and link information
Page Age Get the page age

9 SEO FireFox Plugins I Can’t Live Without

I decided to document my last event. don’t know what this falls under, but hopefully it helps others.

so lets get started

the event

I was contacted by the client to do a live webcast for a horse show. I would setup everything needed prior to the event, and also come to the event location for a day and a half to insure everything ran smoothly, was setup, etc.

the setup

We used fms installed on a media temple extreme server. The extreme server was to be used initially because we didn’t know how many users would be watching. if and when traffic to the feed started to elevate we’d contact mt to upgrade to a nitro server.

for broadcasting we used flash media encoder

day one

I arrived at the event to learn we still needed to pick up some of the equipment to access the video feed. We got to circuit city (no fry’s nearby) and decided on the WinTV-HVR-950 model 1145 from hauppauge.

after arriving at the event, this is the day before the actual event, we found out the a/v people wouldn’t arrive till the morning of the event so the day was pretty much spent making sure everything was running, installed, etc.

day two – seehorse politics

day two started off badly to say the least. the company doing the video “seehorse” apparently had some issues because they had originally wanted to do the web feed. this lead to us receiving the feed used for internal broadcasting which completely sucked and had no audio, we dealt with this the best we could.

luckily they also had brought in outside video people to assist with the filming of the event. Namely Thomas “Rick” Rommarito. I can’t say enough about Rick Rommarito. Rick saw our feed and told us not to worry. Rick ran us a direct feed to the video, audio, and was a great help. keep in mind that seehorse video’s staff had pretty much told me and the company that had hired me to blow off (with out saying it of course). we we’re pretty much ignored by the seehorse video team. so thank you again to Thomas “Rick” Rommarito for everything he did. He is the only reason we got a good video feed or audio. If you found this article while searching on seehorse, please don’t use them, contact rick directly at : rickbommarito[at][sbcglobal][dot][net]. Very knowledgeable a/v guy.

day two – the server

so half way through the day we finally had a good video and audio feed, after a few compatibility issues and tweaking we we’re up and running. here are screen shots from both fms admin, and top to show you how we’re on resources. the cpu usage for the fms process is wrong, so look at the actual cpu usages. for any one interested we maintained about 220-230 concurrent users throughout the day. once that number started to increase is when the server started having issues (end of day). I’d say keep users at about 200 concurrent if your going to be on an extreme.

early in the day we realized we’d need to upgrade to the nitro asap. I contacted mt, and was told we could be upgraded by the following morning. near the end of the day when i contacted mt again for an eta we found out all nitros were sold out for at least a week.

*update* – turns out there was a communication issue. one of the techs had a nitro on standby, but the night shift wasn’t in the loop. found this out the following week

edgecast to the rescue

with our upgrade plan now shot we needed a new host. after limelight told us there’s a 7 day wait, and Akamai seems to have outsourced their entire operation. speaking with someone at Akamai was just pointless, not to mention I didn’t hear back from a sales rep for two days, and that was also outsourced.

*update – I received a call from their guys two weeks later, asking what type of service i want!*

Edgecast was a different story, they knew we we’re 3 steps from being screwed and had us up and running within hours, I’ve already spoken on them in a previous post, but i wanted to make sure i mentioned them again. it normally takes a few days to get everything setup, but james seriously helped us out.

day three

day three had us with a brand new video feed, a new feed to computer setup, and edgecast streaming.

first off the edgecast streaming is great, any future events will use them, screw maintaining our own fms servers.

by the end of the day everything was finally running smoothly, and i got a flight heading back home

issues
our biggest issue was video, which luckily for us Thomas Rick Rommarito handled that.

next up was our server, we’d previously spoken to mt to make sure the nitro was ready on standby. *update – see previous update * I can’t be mad at mt though, as they have proven over and over again to be rock solid and caring. they even contacted me twice after about the nitro issue. edgecast was amazing. their support, network, and customer service was amazing. both myself and the client we’re happy with them, and now the client will be rolling out an entire service using the edgecast system, which leaves me happy to build with their system again.

after those two major issues the reoccurring issue was stream quality. horse shows are very motion intensive, and although we had a good feed, we didn’t have good lighting at all. i recommend any hauppauge product, because I use them at home, and every other encoder tested lacked the features it had. we tryed a few different one’s at the clients request, but in the end always went back to the hauppauge.

to overcome some of the lighting issues i used hauppauge’s built in features to adjust the feed which helped produce a better feed. the client had purchased a number of other encoders because he wanted to split the video / audio with another device. I’ll never again use a pyro product. once we started using the pyro adapter it created nothing but issues.

so once we had the hauppauge device back as the video source i was able to tweak settings and get a decent quality stream that had little lag, good motion, and decent to to good quality. I found the sweet spot to be at a 380 bit rate. the client wanted clear motion, but also needed the horses and people to be of good quality also staying around 380 offered the best motion and quality without being to bandwidth heavy. we broadcasted at a custom 16:9 resolution.

another issue was that the client had an outsource team that developed, and maintained the broadcast site. any changes had to go directly through them. they’re a good group, but often the need to contact them for changes, as well as time differences we’re frustrating. I tried to use the player the client already had in place, but in the end just hacked out a quick and simple player when i finally got completely frustrated.

in the end

so what did i learn? motion intensive events are a different experience than non motion intensive ones. don’t use pyro products. use edgecast, the quality of our stream when using their network was worth it.

I’ve normally ran our own fms server for broadcasts events. I won’t ever again, the cost and benefits of edgecasts service just makes setting up and maintaining a fms server pointless. It’s less of a headache, and you don’t need to monitor anything. In the end the cost of using edgecast is less when you add the man hours, maintenance, headaches, chain smoking, etc.

media temple dv extremes are good for around 200-210 concurrent users, after than your gonna start feeling it.

also something I realized : widescreen formats (16:9) are better then 4:3 for live streams. I found that when using a wide screen format the viewing experience was alot better, especially at smaller sizes than traditional 4:3 format. i was able to reduce the viewer and stream size by going to widescreen, and the viewing experience was actually better.

Server Resources

that’s all folks

yes i know there’s more than goes into live streams, but this is just me giving a brief overview from an experience. hopefully this helps someone, and wasn’t just good intent, bad execution

Sid

If you’d like to see the event streams recorded check out : http://www.iequine.com/

**this is a rant, sorry but needed to vent**

so I’m bout to doze off when i get a message on skype. A clients dv has been locked up since 4pm. its now 11:20. I have no idea what happened, hell I’ve only been on it 3 times. but come on its an extreme thats only running rails and this is the same server mt lost the data on.

why would it now be maxed out cpu and memory wise when it handled the rails site for 2 weeks, and was able to take a licking from fms and keep going….

no friggin clue, but i may start recommending someone else if this becomes a trend

this is the first bad week in over 6 years of hosting with them. I still have faith, but can’t really defend them when the clients pissed and right.

I guess I’m just frustrated. some of the issues have nothing to do with mt. but all the client sees is the mt issues, regardless of it there were video issues on location, or brief communacation issues with the team, or hell me not seeing that they would get the traffic they got and getting a cdn to begin with. All they see is that we had an issue it got fixed, and then they feel mt dropped a major ball. to top it off the not migrating the database thing completely broke the camels back. Now their borderline back to windows streaming.

this was the clients first time using flash instead of windows streaming. I’m frustrated and sad to see them go back to windows, even though this broadcast has gotten the highest reviews from users than any other they’ve done. many of which have out right said : the quality and ease of this years broadcast was better than the vegas event, etc. It’s like you show someone a better way because you want to help them. But i also understand that to them windows is easy and familiar. not to mention their pissed off because our feed is now branded with a competitor, and theres nothing we can do.

In the end i guess the blame rests on me. They had on idea what type of traffic they would get, instead of starting small and having the nitro as a back up, I should have just put them on the cdn to jump, might have cost more from start, but then we’d never have had these issues with mt.

its been a long day, hopefully i’ll have some fun code posts tomorrow

We received a letter from a lawyer represeting the CRIA, they were threatening with legal action and we need to start blocking Canadian traffic because of this.

Thanks for your understanding, and sorry for any inconvenience.

the feedWordpress plugin has been running smoothly on the site I’m using it for.

A few things I’ve noticed.

• feeds with ads in them tend to cause hangs when processing.
• I kept having issues with some post having permalinks to the main site, instead of the site the post was originally from. removing popularity contest corrected this.
• if a site has a post with alot of images, the script will crash. it’s understandable, but still worth mentioning. I had to disable a good site because they had a new post that was huge and image heavy
• use as few feeds as possible
• you should have a decent amount of traffic if you use automatic feed update.

there you have it, my non-review review of the feedwordpress plugin.

check it out :
http://projects.radgeek.com/feedwordpress

David Seah has a good walk through for optimizing your dv server for wordpress

I’m running a (dv) Base system from Media Temple, which is their 256MB version. This is their lowest-end server configuration, which is actually a virtual server sharing the same hardware.

As my blog has grown in popularity, when I switched over I got enough traffic to overwhelm the server. It turned out that the base configuration isn’t tuned to work within the guaranteed 256MB of memory allocated. It’s configured to run on a considerably larger server, and grabs extra memory as needed if it’s available from the global memory pool. This is all fine and dandy when memory is available, but if it ISN’T, the server will start to chug.

After making the following tweaks to my server setup, one of my articles was dugg and the server was hit by 2750 pageloads per hour for a few hours. This is a pretty significant amount of traffic, and the server had no problems staying up. In fact, it barely broke a sweat. Keep in mind though that in addition to the server tweaks below, I also had optimized my wordpress installation with WP-Cache, which reduces server load significantly. You can read more about this in WordPress and Shared Hosting.

check it out :
http://davidseah.com/notebook/configuring-media-temple-dedicated-virtual-server-for-wordpress/

it’s been a day or so and my technorati fix is working alot better than i thought it would be. everytime I check my pings it’s never more than an hour since the last one. So I’ll consider that successful

**update**

I’ve been getting more traffic, but for some reason the pings aren’t going through? I’m changing th3e inval, so maybe this will fix it.