reference: ultimate tag warrior tag clouds
18 Jul
just a reference point for myself
11 Dec
So after 3-4 days of research and study I’m compiling a list of links that helped me finally understand couchdb. Still bunches to learn, but hopefully it will save others from 4 days of googling.
The very first thing you should read is the work in progress online couchdb book:
Relax with CouchDB [http://books.couchdb.org/relax/]
The following sections are grouped by what I left the article better understanding. They may cover other areas, but then again more knowledge leads to better understanding right?
Just in case you don’t understand json:
http://webt.wordpress.com/2007/10/01/json/
fyi: hashes
Depending on what language your coming from you may know hashes as arrays or associative arrays. When they say reduce returns a single value, they are referring to the hash value it returns (scratched my head for a while)
If your coming from php an easy way to connect the dots is to think of how serialize creates a string that represents your object. Only in couchDb this is a json string
I had my eureka moment here:
http://www.ibuildings.com/blog/archives/1291-Some-thoughts-on-CouchDB.html
Just in case you didn’t eureka:
http://rrees.wordpress.com/2008/03/09/couchdb-querying-data/
Damien Katz explains more on couch’s mapreduce ( check the part 2 near the end as well ):
http://damienkatz.net/2008/02/incremental_map.html
In case you didn’t know mapreduce isn’t something couch invented, you can learn more about mapreduce below.
explains the mapreduce method in detail:
http://code.google.com/edu/parallel/mapreduce-tutorial.html#MapReduce
the mapreduce white paper:
http://labs.google.com/papers/mapreduce.html
mapreduce lecture (didn’t watch, but it was recommended by google, so why not):
http://www.youtube.com/v/-vD6PUdf3Js
blog db example / couchdb “joins”:
http://www.cmlenz.net/archives/2007/10/couchdb-joins
user permissions system example / offers rdbms comparison:
http://kore-nordmann.de/blog/couchdb_a_use_case.html
aimee’s 8+ couchdb on rails series (links to part 1, but your good from there):
http://aimee.mychores.co.uk/2008/09/07/post/320/
Ricy ho’s overview:
http://horicky.blogspot.com/2008/10/couchdb-implementation.html
Related discussion where btrees are further discussed (as well as some decent bantering):
http://www.reddit.com/r/programming/comments/792hf/couchdb_implementation/
Ayende Rahien has an indepth series on couch db called: reading erlang
http://ayende.com/Blog/archive/2008/09/24/reading-erlang-inspecting-couchdb.aspx
http://ayende.com/Blog/archive/2008/09/24/more-couchdb-reading-btreelookup.aspx
http://ayende.com/Blog/archive/2008/09/24/more-couchdb-reading-btreequery_modify.aspx
http://ayende.com/Blog/archive/2008/10/04/reading-erlang-couchdb-from-rest-to-disk-in-a.aspx
http://ayende.com/Blog/archive/2008/10/04/erlang-reading-couchdb-digging-down-to-disk.aspx
http://ayende.com/Blog/archive/2008/10/06/reading-eralng-couchdb-streams.aspx
Use it damn it! get planning, hacking, pop locking and start playing with couch
Sources not referenced already:
10 Dec
This is imho the best guide for shell scripting. I’m always referencing it when writing a shell script ( was it -n or -z?)
From: http://linuxcommand.org/writing_shell_scripts.php#contents
With the thousands of commands available for the command line user, how can you remember them all? The answer is, you don’t. The real power of the computer is its ability to do the work for you. To get it to do that, we use the power of the shell to automate things. We write scripts.
Scripts are collections of commands that are stored in a file. The shell can read this file and act on the commands as if they were typed at the keyboard. In addition to the things you have learned so far, the shell also provides a variety of useful programming features to make your scripts truly powerful.
What are scripts good for? A wide range of tasks can be automated. Here are some of the things I automate with scripts:
Check It Out:
LinuxCommand.org: Writing shell scripts.
9 May
I have a friend I’m teaching foundation security to. This post is for him, but also as a protest to some of the materials I’ve found when looking for reference material for him.
Security at it’s simplest form is common sense. ask yourself, how can I make sure I get exactly what I want? How do I make sure I only give what I want. One article mentions xss attacks, and only says prevent them. Why? Thats the question alot of people have when starting why? So why not teach them how to do it first?
How to avoid sql injection / xss, and other misc attacks.
As mentioned this is part rant, part helpful. I’ll explain the following tips and why / how you do it.
first off let me say I’m by no means a security god. Actually I’m not even an advanced user. Sad as it is maybe to say: I’ve never used pear. that said, the majority of attacks / exploits can be easily avoided. Why? because the majority of attacks on the web don’t come from hackers they come from script kiddies. We can be lax with our own stuff ( like this blog ), but any application you build for a client should at least have the basics.
Enough ranting now to the meat and tators…. I’ll keep everything short and sweet. fyi – this is pretty much a brain dump, so prob not in “good form”.
why do we use the _once functions?
if you have a file that loads another file, say index.php?get=/calender.php
what happens if someone changes get to /index.php? yeap your suck in a loop, unless you use require_once / include_once
simple huh?
State changes
Your first question is prob, what the hell is a state change? a state change is simply any change, anytime you change something, whether in the db, a file, an upload: it should always use post. why? Post can be hacked yes, but it’s harder to hack post.
imagine we’re using an online game
ex: update=1&user=87897&add_money=8.
so any user who can add will know: hey i can change add_money to 100 and gain 100 points. On top of that any user can now see all your get vars. Why does that matter? The less they know about your vars, the harder it is for a kiddie / developer to exploit it?
why else? It makes it easier to validate changes. Why? Honestly I don’t even remember why right now, but hopefully you won’t hold that against me
all users are evil
I know kinda overzealous, but you need to have this mindset, why? users will accidentally mess up your system every chance they get. And script kiddies love telling you how l33t they are if they do something as simple as figure out how to make a game page display a different page.
as for making a game page display a different page, honestly: who cares ( yes that was me venting). But in order to prevent accidents, or worst kiddie hacks, control everything! I’m not saying make your app so restrictive that users hate it, but – actually an example would fit best.
today the team made a new flash widget that requires user data.
ex1: pass user data as flash vars, then use loadvars to pass update to server
or more secure
ex2: use loadvars to recieve user data from server, and then pass update to server with loadvars.
in this example we could have honestly used either way as the update script validates all data and any real changes are driven off the database, not the state change, but you get the point. Its one less thing worry about if a user decides to try and change the vars passed, and also one less file to update if we change something.
Users will enter strings when numbers should be entered, upload swf’s when you only want images – you get the point. And the point is validate, and whenever possible take the control from the frontend and move it to the backend.
Be as lazy as possible
I say I am a smart lazy person. building 30 different files takes more time than building one file, and using includes, or a template structure. Pretty lazy huh? but also easier to update and more secure. The more files you have that each have their own independent / copy + pasted code, the more opportunities you have for a slip up. Make one file, and let it handle the logic. You’ll have more freetime, and get more sleep. Or maybe you’ll just spend that time working on more projects. See being lazy is a good thing, but only if done correctly.
we can take this a step further and say why even ftp into the server, it’s so time consuming. why not just build a backend that not only manages your files, but controls access to them – Thats more of a teaser than anything, but try it out, you’ll like the results.
typecast whenever possible
imagine we’re using an online game
ex: update=1&user=87897&add_money=8
ok so what if someone changes add_money to a delete statement, or attempts some form of sql injection. whats the simplest way to defeat it? $money=(integer) $_GET[' add_money'];
Yeap one simple change is all it took to defeat the sql injection. why? Typecasting is basically a way of forcing something to be something. huh? if i want a value to always be an integer, i use (integer). If I want a double i use (double), string (string).
Yeah it’s that simple. the only issue i’ve run into is that you can’t use typecasting in defining function / method params. huh?
ex: function foo((integer)$f=0)
that will cause an error, but you can do
ex: function foo($f=0){
$f=(integer)$f;
}
Make sense? of course I can’t force something to (mocha frap with extra mocha) $coffee, but thats life. good now on to more, or learn more about typecasting
Validate, validate, validate
using typecasting is great for numbers, but theres other ways to validate your data. the best and most powerful being regular expressions
ex: preg_replace(‘/[^a-z0-9]/i’,”,$value);
The above regex replaces any non alpha numeric characters in value. spend sometime getting comfortable with regex as its an extremely powerful and useful feature. Not just for validating data, but regex has many other uses as well.
Be a neat freak, or cleaning your sql
By now you understand sql injection, if not
ok so now we all understand it. basically its a cool way of saying, someones trying to make my query do bad things, but saying it like that would make me should like a user, so we say sql injection and confuse the heck out of clients :p
we just saw how to prevent one form of sql attack. now lets see how we can handle preventing them at the query level.
Whenever data is sent to your db it should always be cleaned. Me I like to make sure both the table, columns are cleaned using a function that makes sure tables / column names follow a standard, and a cleaning function for actual data. Why? When developing an app from scratch you normally have freedom over how tables, columns are named. I prefer to keep all tables and columns lower case, and only allow _ as a special char (non alpha numeric character). what does it look like?
ex: //convert name to proper db format
function dbProperObjectName($objectName){
//if you want to use caps is table / column names then please uncomment this
$objectName=strtolower($objectName);
return @preg_replace(“/([\\x00-\\x2d\\x3a-\\x40\\x5b-\\x60\\x7b-\\xff{$this->mSystemDatabase['restricted_chars']}\\x2f])/e”, ‘_’, $objectName);
}
You can ignore the {$this->mSystemDatabase['restricted_chars']} thats some carry over from the db class. If you don’t understand what heck that says I’ll explain. first I’m changing $objectName to all lowercase, if it’s not already. then we’re using a regular expression (regex) to clean our string of anything thats is not a letter or number and replacing it with _. why does this matter? because if for any reason our table name contains a sql injection, when ran it will only return nothing. why? because if $objectName was SELECT * FROM HOME, it will now be select___from_home. which will return nothing because select___from_home isn’t a valid table. See and you thought cleaning wasn’t fun.
Ok you do windows, but what else?
As much fun as cleaning a table name maybe, we really need to make sure our data is safe. why? ummm because I say so. There are many reasons, ranging from controlling content, preventing xss, sql injection. But I like to think you’ll do it because users are evil ![]()
ex: //strip bad things from a string you plan to use in a query
function dbFriendlyValue($value=false,$fixNewlines=true,$allowedTags=[pass your list of allowed tags here]){
//if no value then just return 0, use this because empty returns false if $value =0
if($value===false) return 0;
//convert to string for checking, this is fine for text / numeric values
$value=(string)$value;
//strip slashes if magic quotes enabled
if( get_magic_quotes_gpc() ) $value = stripslashes( $value );
//clear white space
$value=trim($value);
//fix \r\n
$value=str_replace(“\r\n”, “\n”, $value);
//clear tags (except allowed) or just use html entities
$value=(!empty($allowedTags)) ? strip_tags($value,$allowedTags) : htmlentities($value, ENT_QUOTES);
//change newlines to <br>
if($fixNewlines) $value=nl2br($value);
//clear any bad sql we might find untested regex
$value=@preg_replace(‘/(insert(\s?)into|\).(\s?)values.(\s?)\(|DELETE.(\s?)FROM|CREATE.(\s?)[datbsetl]{5,8}|alter.(\s?)[datbsetlcoumn]{5,8}|drop.(\s?)[datbsetlcoumn]{5,8}|update.\s?(.*?).\s?set|alter.(\s?)[datbsetlcoumn]{5,8})/i’,”,$value);
//add slashes
$value=(@mysql_real_escape_string($value)) ? @mysql_real_escape_string($value) : addslashes($value);
return $value;
}
woah what the hell was that? it was me doing the windows and the oven. lets break it down
when calling the function we pass the value, whether to fix newlines ( default : true) , and the string containing allowed tags if any.
next we make sure we have a value to clean, if not return 0, just in case the function is being used to create a sql statement. we check for magic quotes because if this value came from a submitted variable and magic quotes is on, it will add slashes. if its on the strip those slashes so we can continue.
i’ll skip trim and str_replace, now we’re at strip tags. php is pretty good at striping tags, but you want another option use htmlentities( $value, ENT_QUOTES)
and now on to our regex. this is untested ( sorry still building the class ), but points you in the right direction. the regex searches the value for any sql statements and strips them. lastly we add slashes to our value to make its sql / db safe.
woah – we’ve covered alot. almost done
setup php right
TURN OF REGISTER GLOBALS! yes thats all caps for a reason. Also disable magic quotes and change the headers sent my apache to hide version / software information. Can’t turn of register globals? try this function:
function clearRegisteredGlobals(){
global $_GET,$_REQUEST,$_POST,$_SESSION,$_COOKIE,$_FILES;
//check if register globals is on – register globals check taken fron drupal installed patch : http://drupal.org/files/issues/register_globals_check-D6_3.patch
//get php ini setting
$register_globals = trim(ini_get(‘register_globals’));
//check ini value
if(!empty($register_globals) && strtolower($register_globals) != ‘off’){
//ok now lets clear the variables set with register globals
//make array of superglobals
$registered=$_REQUEST;
$registered=(!empty($_POST)) ? array_merge($registered,$_POST) : $registered;
$registered=(!empty($_GET)) ? array_merge($registered,$_GET) : $registered;
$registered=(!empty($_SESSION)) ? array_merge($registered,$_SESSION) : $registered;
$registered=(!empty($_COOKIE)) ? array_merge($registered,$_COOKIE) : $registered;
foreach($registered as $var=>$void){
@unset($GLOBALS[$var]);
}
}
}
Hide everything
hide everything – that simple. if a folder, file, etc doesn’t / should be seen hide it. How? well if you use the .inc file extension like me, configure apache to handle .inc files with php. another option, or added protection: use htaccess to prevent access to .inc files, this will not effect your scripts, just web browsing.
In addition to hiding your inc files, don’t allow access to directories that aren’t needed to view your site. so you images directory should allow access, but your lib, class, or inc folder shouldn’t.
Important files (db config) should be in a directory outside of your hosting directory, but if you name it .inc, or .php and follow these directions you should be ok.
Lastly – turn off directory browsing.
Control access
ever part of your site should have an access level. So areas like your home page, public areas would a level 0. areas a users settings page would be a 1 (making sure only the user can access it of course), and your admin area – thats another story. Your admin area is the heart / backbone / investors dream of your site. That said protect it! all users in your admin should have an access level, and different parts of admin should have different access requirements.
ex: moderator – login, see’s flagged post area, does not see links to other areas, can not access other areas. manager: login, sees users, can add or remove users, but can not access critical site areas, and can not add a user >= his level. Admin can do almost everything, and lastly: rot – your root account can be named anything, but only allow one account full control over the system.
so quick review: users should only be able to see and access areas within their permissions scope, users should never be add / give users permissions >= their permissions.
Lastly, track everything your admin users do. You can go as far as adding an approval system for changes, tying your backend to svn to undo / redo changes, it’s pretty much up to you and the project / budget.
using sessions
sessions are like raymond, everybody loves them. But if your depending fully on php sessions you should make some changes. There should only be 1-2 cookie and session variables sent ( you can also send session id with get ), everything else should be handled internally in your application. Which means session / user validation, tracking, and variables.
misc
Read the php manual, you’ll find lots of good advice / functions in the classes.
Are we done?
Yes, hopefully someone gets something out of this, and I kept my promise of short and sweet. a quick google is all you’ll need to learn more about a subject. so right click -> search google
Gotta question, feedback, or recommendation? leave a comment
Cheers Sid / Greg
31 Mar
Came across this today. rocket99 has an indepth reference cover all the *nix basics
11 Mar
Before you start
Before starting anything you should be sure that you have the developer tools package installed. If your not sure, then you don’t. No worries: submit a request to install the developer tools package and you’ll be ready in about 45min to a hour.
Get started
first login as root
next download electroserver:
wget http://www.electro-server.com/downloads/builds/ElectroServer_4_0_4_linux.rpm
next install the rpm:
rpm -iv ElectroServer_4_0_4_linux.rpm
You should now have a /opt/ElectroServer_4_0_4/ directory. browse around it and we’ll continue from there.
I had no issues up to this point, but feel free to post any problems and I’ll try to help.
Testing the install
in the /opt/ElectroServer_4_0_4/ you’ll notice a few files. to test your install run the ElectroServer command
If your lucky you’ll have no errors or issues getting the server running. I encountered an issue related to memory available. a quick reboot of the server fixed this.
Configuring the server
For this tutorial we’ll use port 8080 for the web admin, and port 9898 for our chat server. if you need to use other ports just follow along. you can also repeat this process to enable additional ports.
we’ll need to enable access to port 8080 and 9898. port 8080 will be used for the web admin, port 9898 will be used for the text (chat) connection.
to open your ports run the following:
port 8080
/sbin/iptables -A INPUT -p tcp --dport 8080 -j ACCEPT
/sbin/iptables -A OUTPUT -p tcp --dport 8080 -j ACCEPT
port 9898
/sbin/iptables -A INPUT -p tcp --dport 9898 -j ACCEPT
/sbin/iptables -A OUTPUT -p tcp --dport 9898 -j ACCEPT
next edit your ES4Configuration.xml which can be found at: <install directory>/server/config. if your following this tutorial on a media temple dv server this directory is /opt/ElectroServer_4_0_4/server/config/ES4Configuration.xml
ES4Configuration.xml your default configuration should look like this:
<?xml version=”1.0″ encoding=”UTF-8″ standalone=”no”?><ElectroServerConfiguration>
<!– This is the name of the node –>
<Name>StandAlone</Name><!– Defines the host and port on which the web server will listen for inbound connections –>
<WebListeners>
<WebListener>
<Host>127.0.0.1</Host>
<Port>8080</Port>
<Ssl enabled=”true”/>
</WebListener>
</WebListeners></ElectroServerConfiguration>
to get our server live we’ll need to give it an ip address, change 127.0.0.1 to an ip that is live on your server
Getting the server running and listening
before we can actually test our install we need to make sure that we’ve placed a crossdomain.xml at the root of the website listening at the ip used. if the ip is shared place the crossdomain file in the root of the default domain. the crossdomain.xml should be viewable at http://<electroserver ip>/crossdomain.xml, replace <electroserver ip> with the ip used in your ES4Configuration.xml.
be sure that you create allows for your running host, and any other domains that will access the server.
now restart electroserver if its already running and test if electroserver is excepting connections using telnet.
test that electroserver is running and able to listen by running telnet <electroserver ip> 8080
telnet should successfully connect. if not try shutting down all electroserver processes and start electroserver again. we’re now ready to get moving with getting our chat server running.
Use electroserver admin
the configuration used in this guide uses ssl, if your not using it then remember to use http where the guide says https.
access admin at https://<electroserver ip>:8080/admin
default user name: administrator
default password: password
once logged in go to gateways and edit the default gateway. change 127.0.0.1 to <electroserver ip>, save and restart the gateway
Test Chat Access
Electroserver by default runs chat on port 9898. restart electroserver and telnet to your electroserver ip, this time testing port 9898. your should recieve the contents of your crossdomain file as a response.
Almost Done
we’ll also need to add electroserver to our server startup. First create a file named: StartElectroServer, next add the following to that file:
/usr/bin/nohup /usr/local/bin/ElectroServer &
exit
Place the script in you opt/electroserver directory with read and execute permissions. Now the final step link your file so it’s called at startup
ln -s /opt/ElectroServer_4_0_4/StartElectroServer /etc/rc.d/init.d/StartElectroServer
thats it, you should now be all set to run electro server.
Also to run electroserver without it closing when your console closes use: nohup ElectroServer &
Got questions? post a comment, or just post a comment to let me know this was helpful.
Sid
25 Feb
This is probably the fastest way to exclude category in WordPress besides using a plugin. This is also the same method that I use for Shortnotes plugin.
28 Jan
There are alot of ascii quick references on the web. I ended up using about 5 of them today only to find out the ascii codes the had listed weren’t correct, at least not when attempting to find the hex codes for regular expressions.
Luckily i found a great ascii reference at acsiitable.com. they the decimal, hex, oct, and html codes. best yet, there all correct.
check it out:
Ascii Table – ASCII character codes and html, octal, hex and decimal chart conversion
Powered by ScribeFire.
25 Dec
this is more for my reference than anything, but I cover the areas i modified to integrate wordpress in to a clients current site. I’m posting this, mainly to keep me from having to compare files everytime i do a wp upgrade. its by no means a solid tut as there are additional site classes / templates I’ve made to integrate the look / site functionality / security that are also used and not mentioned / documented here. as i said it’s written for myself, so i’m assuming you understand php / wordpress
wp-config.php
on the very first line of config.php include either the db config file, or the site wide control include (file that handles all of the general includes, etc)
set wordpress db config to use the globally defined vars. ex: define(‘DB_NAME’,SITE_DB_NAME);
any additional wordpress only includes should be included at the end of this file.
i also added in a secondary check to see if the current site user is logged in to wordpress, you can gt the current user as follows :
$the_user= wp_get_current_user();
$user_ego=$the_user->user_login;
if the current user is logged in to the actual site, but not into wordpress, i attempt to log them in using a login function in my custom site to wordpress connector class.
wp-db
bottom of file change :
if ( ! isset($wpdb) )
$wpdb = new wpdb(DB_USER, DB_PASSWORD, DB_NAME, DB_HOST);
to
if ( ! isset($wpdb) ){
global $wpdb;
$wpdb = new wpdb(DB_USER, DB_PASSWORD, DB_NAME, DB_HOST);
}
this will fix issues you run into when including the wordpress connections into a site / custom class.
Admin Area
open wp-admin/admin.php, setup any globals that are needed to require a vliad user for an area. ex: define(‘REQ_VALID_USER’,true)
wp-config will be included immediately after this. look for the line that says $posts_per_page, and add the something like following to limit access to the actual admin area:
global $current_user;
wp_get_current_user();
if(!empty($current_user)){
if(intval($current_user->user_level)< =WORDPRESS_LIMIT_ACCESS){//this var should be defined in your site wide define file
global $_SERVER;
$url=$_SERVER['REQUEST_URI'];
if(strpos($url,”?”)!==false){
$url=substr($url,0,strpos($url,”?”));
}
if(strpos($url,WORDPRESS_ADMIN)!==false && strpos($url,WORDPRESS_LIMIT_ADMIN_STR)===false){
header(‘location: ‘.WORDPRESS_LIMIT_PAGE) ;
exit;
}
}
}
the above code will check the curent url against the allowed urls for general users in admin, as well as redirect them to they limit page (write post) if they aren’t at the proper level. everyone else will be able to use the admin according to their level.
limiting post categories
either wp-admin/admin-functions.php or wp-admin/include/template.php
look for : function write_nested_categories( $categories ) {
limit categories as needed for general users, but be sure to leave an unmodified version for admin users.
remove nav for general users
replace : require(ABSPATH . ‘wp-admin/menu.php’); with
if(intval($current_user->user_level)>WORDPRESS_LIMIT_ACCESS){
require(ABSPATH . ‘/wp-admin/menu.php’);
}
customise nav header
wp-admin/admin-header.php
peace, sid
2 Dec
Tribal Fusion is a great service, but alot of people aren’t fans of their tribal fusion direct service.
This list may be somewhat limited as it’s only the domains I’ve encountered using tribal fusion, I’m sure theres others, and if you know some please post them here. use your own judgment when banning domains, as some of the market domains also server via tf direct.
List Of Tribal Fusion Direct Domains
I’ll update this list as more domains are found
imvu.com
zwinky.com
smileycentral.com
aol.com
atdmt.com
msn.com
lasikplus.com
microsoft.com
onlinerewardcenter.com
starware.com
screensaver-horoscopes.com
cursormania.com
smileycentral-intl.com
freepay.com
freeipods.com
screensavers.com
smileycentral.com
yourgiftexperts.com
YourGiftCards.com
puzzlepirates.com
onlineprizecenter.com
bestfreerewards.com
classmates.com
reunion.com
insweb.com
consumerincentiverewards.com
inklineglobal.com