Tag Archives: python

couchdb: how to do "joins" in couchDb

10 Dec

While continuing my trolling for all things couchdb, I came across this tutorial. Christopher Lenz uses a blogging system to explain how a couchdb database could be setup and how to perform a join (or something similar). Better yet the reccomended join method comes from the couchdb creator himself :)

From: http://www.cmlenz.net/archives/2007/10/couchdb-joins

I’ve been playing more and more with CouchDB lately. After putting together a Python library, I worked on a brand new included HTML/AJAX interface. Now I’m starting to dive into the Erlang code, which is my first serious encounter with Erlang. In particular, I started a branch that aims to replace the HTTP server underpinnings with Bob Ippolito´s not-quite-released-yet Mochiweb library.

Despite all that activity (and past experience with the conceptually similar Lotus Notes), the correct approach to designing applications “the CouchDB way” isn’t always obvious to me at this point. Just today, there was a discussion on IRC how you’d go about modeling a simple blogging system with “post” and “comment” entities, where any blog post might have N comments. If you’d be using an SQL database, you’d obviously have two tables with foreign keys and you’d be using joins. (At least until you needed to add some denormalization.)

But what would the “obvious” approach in CouchDB look like?

Check It Out:
about:cmlenz – CouchDB “Joins”

good read: Fighting Technological Indulgences

2 Dec

That’s a nice one! Wonder if I should use Ruby, Python, or maybe even Perl to build it? As gearheads we tend to jump to the technical details with little afterthought, but should you even be building that feature in the first place? How do you make that call?

Whether you’re a one man team with a shoestring budget, or a financed startup with some runway, managing the feature backlog is more of an art than a science – it’s always full of vaguely formulated ideas and it’s constantly changing. Managing these priorities is as hard as it is, and if you have a team, it gets even harder: how do you make sure that everyone is aligned with the vision and the roadmap?

Fighting Technological Indulgences – igvita.com

quick start: web project management

13 Nov

So I’m trying to give a friend a few of the basics they’d need to know for managing a web project. Not a bible, just a quick read to hit the ground running. Here is what I came up with.

Your team:

designers – creative artistically, creates look and feel, communicate, produces things you can see
developers – creative technically, creates applications foundation, communicate, leave alone for extended periods, produces things that are used, but not always seen.
cto – head geek, usually smarter than you.
ceo – head head, smile and nod – might be cool
cfo – head money – nuff said
creative director – awesomely artistic, likes coffee
techincal director – almost head geek
vp anything – has friends, probably knows what their doing
assitant – love them, bring them coffee, and be really really nice to them
project manager – you, makes sure designers / developers are making progress, middle man between client, developers / designers, and everyone else (maybe)

Basecamp:

where you document everything. Covers your ass. Anything changes, legals, documents, or client communication happens here.

Cover your ass use BC

Deadline:

in house milestone / deadline=client review milestone / deadline- (1 – 2 weeks)
client milestone / deadline = milestone / deadline + (1 – 2 weeks)
client review milestone / deadline = milestone / deadline – (2-3 weeks)

Terms to know:

fubar – we’re fucked
need more time – your fucked
great – needs review
fun – expect delays
breach of contract – see fubar * 2
contract – covers your ass
invoice – pays your ass
server – where everything runs
ui / client side – where everything is seen
php, ruby, python – what developer use to make things run
css / stylesheet – what designers / developers use to layout what you see
ajax – those cool effects
flash – more cool effects
flv – youtube video
harvest – where everyone tracks time
bc – where you track clients / team
skype – online meeting
delay – no sleep
billable hours – what you can get paid for
over hours – what you might get paid for
favor – 5/1 to get your account established
hail mary -  a favor that empties your favor account
wireframe – lines that show how something will look
mock up – test design
voip – the phone
im / aim – chat
ping – contact

And there you have it, my quick read, more web filler.

sid

Building somethng scalable: language / frameworks aka use ror or php

13 Nov

When I first started this experiment I planned on using a custom php framework. Recently I realized that its kinda pointless to attempt to do something like this and at the same time lock myself into something that may not be the best solution….

enter google, research, testing, and little sleep. Result: codeigniter was the best choice. Huh? Here’s why.

snakes not on the plane

I should say from jump: If I knew and had the time to learn python / django it would have been the winner, sadly I do not.

ruby, rails, hype

I’m a fan of ror, and of course when I decided to take what I’ve created so far and migrate it into something usable I of course thought of ruby on rails. Sadly I saw more than a few hurdles.

First off ror doesn’t play we’ll with others, meaning you shouldn’t run rails in a shared environment. The whole concept behind this experiment is a small start-up with a shoe string budget and 2-3 websites / apps. Yes its possible to host multiple rails apps on one server / vps, but it’s not recommended. There is of course passenger, but that leads into the next point. php +1

Ror is a resource hog, there I said it. When compared to php, rails is more resource intensive than php. Of course the answer is to optimize and scale out, but remember I’m trying to keep the monthly hosting budget under 100-120 bucks (yes seriously shoe string). So php was the choice here.

So far php seems like the best choice for what I’m trying to do, but I needed more than just a few issues I could work around. You can’t work around speed and optimizing. ruby is faster than php via command line, but ror is not amazingly faster when used via web. When you add an opcode cache into the picture ror gets is butt kicked hands down. Of course this is comparing a language to a framework, enter codeigniter.

Codeigniter still out performs ror with an opcode cache. That is just one optimization and php shows drastic improvement.

I wasn’t ready to abandon ror yet, simply put: why use an imitator if you can have the real deal. Codeigniter is a great framework, but it makes more since to actually use rails right?

In the end the answer is no. Rails has a higher level of maintainability out the box, but does less code, easier maintenance, and of course the ror cool factor out weight slower speed, higher cost to deploy, and fewer production optimizations? No.

MVC while not as strict in codeigniter, is there. That combined with OOP will make using a php framework easier to maintain, not ror easy, but easy enough for production imho.

framework, shamework

So we now know why php won over ruby, but why pick codeigniter? I originally started out writing a custom framework, which is always fun to do from time to time to push yourself, but in the end you see the downside and benefits of doing so. Since this entire idea revolves around a small start-up we also need to take development time into consideration.

A custom framework takes alot of trial and error, coding, and recoding, and still more coding. Using a php framework I can reduce the time it takes to get up and running, while still building what I want and need. Yes there’s some overhead compared to a custom framework, but in the end a start-up isn’t yahoo. You should be back at the planning table long before you reach yahoo numbers. Thats not to say you shouldn’t plan for massive growth (hence scalable), just that traffic / users / data on yahoo’s scale is far beyond my scope of experience and even I know its another elephant to eat.

When comparing php frameworks, i was originally looking at cakephp, but after some research found it to slow and with to much overhead. There are other frameworks, but only codeigniter and cakephp met my needs. 2-1=1 (or less than one mr. V), so codeigniter was the winner.

One thing I should note is how poor most of the php frameworks I looked at perform out the box. Yes rails (and merb) are better out the box. Luckily a few simple optimizations speeds things up.

all your base are belong to theory

So far everything has been pretty much all theory, research, and testing. No worries, the meat and potatoes are coming shortly.

the next few posts will cover setting everything up(centos5): memcached / memcachedb, ngninx, apache, varnish, mysql, s3 + rsync, varnish -> s3 relations, etc.

If you feel like following along head over to http://linode.com and setup a few machines, or setup a team in vmware.

Worth Reading

Here are few articles that I came across while researching, some are more on topic than others, but all have some value.

PHP vs Java vs Ruby:
http://www.cmswire.com/cms/industry-news/php-vs-java-vs-ruby-000887.php

Ruby vs PHP performance (cli)
http://izumi.plan99.net/blog/index.php/2008/01/17/ruby-vs-php-performance/

Ruby on Rails Fans
http://shiflett.org/blog/2006/feb/ruby-on-rails-fans

The performance test of 6 leading frameworks
http://www.alrond.com/en/2007/jan/25/performance-test-of-6-leading-frameworks/

Joyent: free accelerators for facebook apps

31 Mar

Facebook, Joyent and Dell have partnered to provide free scalable, on-demand infrastructure from Joyent to Facebook developers. Joyent’s Accelerator on-demand infrastructure (peered with Facebook’s datacenter!) provides the very best load balancers, routing and switching fabric, x86 servers and storage from Dell. Facebook developers can take advantage of Joyent Accelerators to quickly launch Facebook applications capable of scaling to millions of users. All for free.
What’s Included?

A Joyent Facebook Developer Accelerator includes everything you need to develop and deploy your Facebook application. You get root access to a virtualized machine that includes all the tools for developing Facebook applications in PHP, Rails, and Python.

Note: We do not currently support Windows, and do not plan to do so in the foreseeable future.

Joyent: Facebook Developer Program

tying a custom sytem into plesk

29 Jul

ok well after stayng up for most of this weekend i’m still gonna end up having to use what i already had yesterday. but maybe my time wasted can help someone out there.

1. plesk’s cmd line subdomain creator does not tie into dns. so after slaving to have everything work perfectly with plesk – this is just another hill to conquer.

2. with the default mysql / plesk setup you have to use grant to add a user

3. there are there tables in the psa database that contain the info needed to display a subdomain in a users subdomain panel.

heres the code that finally worked for me

function processOnServer($clientName){
global $baseDir;
//make our password using the user name and adding some md5 salt to the tail
//each password must be unique, as it's how we'll find the ids later
$pass=md5($clientName);
$passLen=13-strlen($clientName);
$passEnd=substr($pass,0,$passLen);
$password=$clientName.$passEnd;
//setup the client dir location
$clientDir=$baseDir.$clientName;

$dbConn = mysql_connect('db_server', 'user', 'pass');
if (!$dbConn) {
    die('Could not connect: ' . mysql_error());
}

$dbName=$clientName."_trac";
//create db
$sql = "CREATE DATABASE $dbName";
mysql_query($sql, $dbConn);

//select mysql db
mysql_select_db("mysql",$dbConn);
//add client user
$sql="GRANT ALL ON $dbName.* TO '$clientName' IDENTIFIED BY '$password'";
mysql_query($sql, $dbConn);
//select plesk db
/*plesk configuration - i'm leaving this till another day
mysql_select_db("psa",$dbConn);
//add account
$sql="INSERT INTO `accounts` (`id`, `type`, `password`) VALUES (NULL, 'plain', '$password')";
mysql_query($sql, $dbConn);
///find the new users id
$sql="SELECT id,password FROM accounts WHERE password = '$password' ORDER BY id DESC";
$findAccountId=mysql_query($sql, $dbConn);
list($newId) = mysql_fetch_row($findAccountId);

//add user to plesk for subdomain
$sql="INSERT INTO `sys_users` (`id`, `login`, `account_id`, `home`, `shell`, `quota`) VALUES (NULL, '$clientName', $newId, '$clientDir', '/bin/false', 0)";
mysql_query($sql, $dbConn);
//find the new new id
$sql="SELECT id,account_id FROM sys_users WHERE account_id=$newId ORDER BY id DESC";
$findAccountId2=mysql_query($sql, $dbConn);
list($newerId) = mysql_fetch_row($findAccountId2);

//add subdomain to plesk
$sql="INSERT INTO `subdomains` (`id`, `dom_id`, `name`, `displayName`, `sys_user_type`, `sys_user_id`, `ssi`, `php`, `cgi`, `perl`, `python`, `fastcgi`, `miva`, `coldfusion`, `asp`, `asp_dot_net`, `ssl`, `same_ssl`) VALUES (NULL, 1, '$clientName', '$clientName', 'native', $newerId, 'false', 'true', 'false', 'false', 'false', 'false', 'false', 'false', 'false', 'false', 'true', 'false')";
mysql_query($sql, $dbConn) or die('Could not connect: ' . mysql_error());
*/
//make config file
$fileName = md5($clientName);
makeApacheFiles($clientName,$fileName.".conf");
makeTracFiles($clientName,$fileName.".ini",$password);
makeAuthFiles($clientName,$fileName.".authz");

//finish the svn and trac setup
passthru("shell_script $clientName $password $fileName");

}

the codes choppy (i’ve been up for over 30 hours – so i’m sure it looks worse to use :) )

the system i’ve built uses a custom frontend app to create, manage subdomains, and do automated installs of svn repos / trac, etc.

if you look over that code you’ll see there’s three psa tables that need fixing to allow your client to also view your subdomains in plesk.

also the ftp fix mentioned on there site won’t work on mediatemple. when i find one i’ll post it.

in addtional to all this. the psa database is only a place holder for the frontend, and in no way reflects the actually server (unless you do everything from plesk)

if you add a db or a dns record you’ll have to also update the psa db.

don’t know if i’m still making sense, off to bed – sid

amazon s3 tools

27 Jul

looking around today and came across a project focused on creating tools for amazons s3. so far the have a python based command line to amazon’s s3 service, as well as a php5 amazon s3 browser.

check it out : http://s3tools.logix.cz

media temple dv install yum and media temple dv simpler svn install

1 Jun

found a tutorial on installing yum on a media temple dv within a tutorial on installing svn on media temple dv

the yum portion :

$ wget http://centos.mirror.vpslink.com/centos-4/4.4/os/i386/CentOS/RPMS/yum-2.4.3-1.c4.noarch.rpm

It has dependencies, so we need get them too.

$ rpm -ivh –nodeps http://centos.mirror.vpslink.com/centos-4/4.4/os/i386/CentOS/RPMS/libxml2-python-2.6.16-6.i386.rpm http://centos.mirror.vpslink.com/centos-4/4.4/os/i386/CentOS/RPMS/python-elementtree-1.2.6-4.2.1.i386.rpm http://centos.mirror.vpslink.com/centos-4/4.4/os/i386/CentOS/RPMS/python-sqlite-1.1.7-1.2.i386.rpm http://centos.mirror.vpslink.com/centos-4/4.4/os/i386/CentOS/RPMS/rpm-python-4.3.3-18_nonptl.i386.rpm http://centos.mirror.vpslink.com/centos-4/4.4/os/i386/CentOS/RPMS/python-urlgrabber-2.9.8-2.noarch.rpm

Now we can install yum:

$ rpm -Uvh yum-2.4.3-1.c4.noarch.rpm

Check if it’s installed:

$ rpm -q yum

go check it out for the full how to:
http://blog.hellm.com/post/5

setup trac on media temple dv or centos with plesk installations

16 May

I covered svn setup on media temple dv and similar centos with plesk setups here : http://elsid.net/2007/05/07/setup-svn-server-on-media-temple-dv-or-centos-with-plesk/

Now i’ll cover the basics of setting up trac on media temple dv and similar centos with plesk setups. Please note this is what worked for me (and should work for other media temple dv setups). Please let me know if i missed something, or any recommendations.

first steps

First you need to request root access and the developer tools package. Once that’s done we’re ready to start.

This setup will use mysql for the trac database and serve trac from a subdomain. I’m only covering the setup of a vanilla trac installation

You can take a look at the trac install guide here : http://trac.edgewall.org/wiki/TracInstall

Preq’s / What you’ll need:

clear silver :

  • download – http://www.clearsilver.net/downloads/
  • setup docs – http://lists.edgewall.com/archive/trac/2005-December/006090.html
  • setup – ./configure, make, make install
  • setup tools :

  • download – http://svn.python.org/projects/sandbox/trunk/setuptools/#egg=setuptools-dev
  • setup docs – http://svn.python.org/projects/sandbox/trunk/setuptools/README.txt
  • setup – sh setuptools-0.6c4-py2.4.egg
  • mysqldb:

  • download – http://sourceforge.net/project/showfiles.php?group_id=22307
  • setup docs – http://blog.spikesource.com/mysqldb.htm
  • setup – python setup.py build, python setup.py install
  • python:

  • already installed
  • mod_python:

  • already installed
  • svn:

  • http://elsid.net/2007/05/07/setup-svn-server-on-media-temple-dv-or-centos-with-plesk/
  • More pre-install steps

    I covered svn setup here : http://elsid.net/2007/05/07/setup-svn-server-on-media-temple-dv-or-centos-with-plesk/

    svn must be rebuilt to work correctly with trac. http://trac.edgewall.org/ticket/3706#comment:10 .

    First we need to edit the svn makefile.
    change : SVN_APR_LIBS = /root/subversion-1.4.3/apr/libapr-0.la -lrt -lm -lcrypt -lnsl -lpthread -ldl
    to : SVN_APR_LIBS = /root/subversion-1.4.3/apr/libapr-0.la -lrt -lm -lcrypt -lnsl -lpthread -ldl -L/usr/lib -lgssapi_krb5 -lkrb5 -lk5crypto
    replace /root/subversion-1.4.3 with what ever is the path in your file.

    now run make clean, then make, and make install

    Almost done. We’ll build swig and then move to our trac setup. in your subversion build folder do the following.

    make swig-py
    make install-swig-py
    make swig-py install

    If you’ve installed all the preq’s, rebuilt svn, and built swig we should be ready to install trac.

    –note : you may need to copy the svn and libsvn folder from /usr/local/lib/svn-python to /usr/lib/python/site-packages. remember copy not move.

    Install trac

    Install Trac
    python ./setup.py install

    Setup Project

    go to plesk and setup a subdomain for trac. Make sure you only select python support.

    now setup a mysql database at the same domain you made your trac subdomain at. Add a user for this database and we’re almost home.

    make the following directorys :
    /var/www/vhosts/yourdomain/subdomains/yoursubdomain/trac/test
    /var/www/vhosts/yourdomain/subdomains/yoursubdomain/svn/test

    setup a project : trac-admin /var/www/vhosts/yourdomain/subdomains/yoursubdomain/trac/test initenv

    setup a test svn for this project : svnadmin create /var/www/vhosts/yourdomain/subdomains/yoursubdomain/svn/test

    make a sub directory in our trac directory and chmod 0777 it, we’ll need this later: mkdir /var/www/vhosts/yourdomain/subdomains/yoursubdomain/trac/plugins
    chmod 0777 /var/www/vhosts/yourdomain/subdomains/yoursubdomain/trac/plugins

    be sure to check the permissions on your svn and trac directory, we need to be sure apache is able to access and run our files.

    go to the conf directory for your subdomain : cd /var/www/vhosts/yourdomain/subdomains/yoursubdomain/conf

    almost done make a file : vhost.conf

    Open this file and add the following, remember to fix the paths to your domain and subdomain:


    SetEnv PYTHON_EGG_CACHE /var/www/vhosts/yourdomain/subdomains/yoursubdomain/trac/plugins
    SetHandler mod_python
    PythonHandler trac.web.modpython_frontend
    PythonOption TracEnv /var/www/vhosts/yourdomain/subdomains/yoursubdomain/trac/test
    PythonOption TracUriRoot /
    PythonInterpreter main_interpreter

    Final Steps

    restart apache : /usr/local/psa/admin/sbin/websrvmng -a -v

    point your browser to your trac subdomain. If everything is working fine you’ll see a trac page. If not you’ll see a 500 error page.

    Got Problems?

    If your seeing an error page you should take the following steps:

    go to plesk >> yourdomain >> log manager >> error log

    copy the portion that contains your error and remove the servers timestamps and ip data.

    do a google on the very last line, this will contain the error message. You’ll likely see a result from the trac website. if not add the word trac before your search.

    If you still see no results from the trac website, go to the site and search the tickets. Still no luck? open a new ticket. Trac is very good with support and the community is extremely helpful.

    References / helpful Links

    http://trac.edgewall.org/wiki/TracGuide
    http://trac.edgewall.org/ticket/3706#comment:10
    http://trac.edgewall.org/ticket/4459#comment:21

    If you need help you can contact me, but i’d recommend using the trac community as i’m not fluent with python.

    happy coding – sid