If you haven’t already realized I’m being cynical, and truly believe there is such a suite, you should definitely keep reading. If you have, you should keep reading anyway. You might actually learn a few things.

There’s no doubt about it. Users can supply your application with data that can break it whether it’s intentional or unintentional. Whether there is a malicious intent, or not. So we certainly don’t want to blindly insert data from the user into our application layer and allow it to inadvertently seep through our code, like in the case of Bobby Tables. However, the problem isn’t that the user’s data is dirty, but that your code has been engineered in such a way that it has developed a data germophobia. This alluding side effect only exasperates your problems in dealing with user-supplied data.

Data Belongs To The User

First, you have to consider that this data isn’t yours to begin with. It’s the user’s data and the user should have control over their own data. However, the code and the application are yours and you, likewise, should retain full control over your code and your application layers. The heart of the problem really lies in the places where those two things tend to meet and the line between where your code ends and the user’s data begins becomes quite blurry. In fact, they may, very well, be indistinguishable at times. So it’s easy to mangle the user’s data and break it just as it can be easy for the user to intentionally — or unintentionally — break your application. In both cases the intent of both the user and the software engineer is unimportant in solving this problem. What really matters is that the solution allows both parties to retain their respective rights in not intruding on the other’s property.

Code Belongs To The Programmer

As a programmer — and especially as a web developer — you’re always taught that user-supplied data is never to be trusted. Sometimes this loosely transpires into “the user is an idiot“, however, that’s just a misnomer. The collective user-base of a web-based application or Software as a Service translates to more valuable knowledge than any two programmers have, combined. This is usually because the engineer works to solve a problem and the user is just looking to get things done without a big hassle. The two have different objectives, but one also has a broader picture than the other.

The notion tends to be that if you soak the user’s data in enough bleach, hose it down with enough Lysol, and dust/vacuum around it regularly, it shouldn’t be a problem. Speaking purely from an analogical point-of-view, of course. However, not all data is created equal, and not all sanitary products are the same. Have you ever accidentally thrown a colored shirt in the wash along with your whites and added bleach? Just as that’s going to ruin all of your whites the same mistake will likely corrupt a bulk of your user’s data (and probably might break your application) as well.

So the real end-goal here is not “this data is a problem“, but more so that “this code and this data don’t seem to get along“. The most obvious solution is to then separate the two and create a clear layer of segregation between them so that one does not interfere with the integrity of the other.

Examples Of Poor Data Sanitization Practices

One common beginner mistake is to think that stripping things from the user’s data might solve the problem that that data imposes on your code. That’s a problem though, because it ultimately means you have removed things the user (for all we know) had every intention of keeping. To me this just means you’ve introduced a new problem (breaking the integrity of the user’s data).

An example of this in PHP is where you want to output user data in your HTML, but you don’t want to allow the user to inject HTML into your output (breaking your application). Thinking that if you just strip all of the (less-than <), (greater than >) characters, from the user’s supplied data, this will keep your code integrity, is a biased view. What about the integrity of the data presented by the user? Why would you assume that the user had the intent of injecting HTML or performing some malicious XSS injection just because their data contained invalid characters that your code cannot accept? Instead, we should find a way to retain the integrity of both our code as well as the user’s data without posing any vulnerability or crippling of the application.

Code

$_POST["input"] = "X < Y && Y > Z";
$output = str_replace(array("<", ">"), "", $_POST["input"]);
echo "<p>The user said: $output</p>";

Output

<p>The user said: X  Y && Y  Z</p>

The above example in PHP is a horrible idea. This is not something you ever want to do. Now the user’s data has lost all its integrity since, in this example, the user simply wanted to present data that states “X < Y && Y > Z“, but your application as rendered their data useless. Think if this were meant to be a post on a public math forum what the impact of your code would be on your user-base.

Lets consider another example…

$_POST["input"] = "
<html>
    ...
</html>
";
$output = strip_tags($_POST["input"]);
echo "
    <h1>User Data</h1>

    <div>$output</div>
";

The output becomes tainted data…

<h1>User Data</h1>

<div>
    ...</div>

Imagine if this were a public forum for web developers and someone were attempting to present some HTML code as a example to some question? Certainly the intent here is to prevent HTML/XSS injection, but that shouldn’t result in breaking the user’s data either. So lets present a real solution to the problem that doesn’t pose yet another problem.

Escaping Vs. Stripping

$_POST["input"] = "X < Y && Y > Z";
$output = htmlspecialchars($_POST["input"]);
echo "<p>The user said: $output</p>";

<p>The user said: X &lt; Y &amp;&amp; Y &gt; Z</p>

This allows us to encode the user’s data to HTML entities that the browser will not confuse for markup. It means the user’s data appears in the browser just as they typed it and there will be no unwanted intrusion of that data into your HTML. Great, now your code doesn’t break and the user’s data retains its integrity. Imagine that, no vulnerability to your application layer and no data corruption! It’s a WIN-WIN situation. The point remains that we do not intrude onto the user’s property and the user does not intrude upon ours. Here both the application layer and the data layer can co-exist in harmony.

What you don’t want to do is escape the user’s data and then store it in it’s escaped form. For example, don’t use htmlspecialcahrs or htmlentities before storing user data in your database. These are meant as transport mechanisms for the document character set, not to be confused with fortifying your SQL against malicious injection. This just means what you have in your database isn’t what the user supplied you with. You would need to take additional measures to unescape the data back to its original encoding in the event you need to perform any actual work on the data, such as searching, transferring, etc…

Instead all you really need to do in order to clearly separate the user data from your application code is make sure you escape it properly (in this case for HTML) before it gets mixed in with your code. The most common approach to avoiding this blurry line, we mentioned earlier, of mixing the two together is to use a templating system. This would be a place where your HTML can live cozily and accept values to be inserted in the template at will, handling the rendering of this view through abstract method. The data you hand it from your modal would than be transported in the proper encoding and the data itself remains unaffected. It’s also, not just, user-data you’re worried about here, but if any of your own application data might break that HTML, as well. So this abstraction is a fitting solution for a common problem.

The SQL Injection Problem

Of course this brings us to the infamous SQL injection attacks that a user can pose on your application by presenting you with data that might break your SQL code if you were to combine that data with your SQL code in the same way you tried to combine the user’s data with your HTML code. The obvious solution has always been to escape that data before allowing your code and the user’s data to mix and mingle together.

$input = mysql_real_escape_string($_POST["userdata"]);

$sql = "INSERT INTO `table` VALUES('$input')";

mysql_query($sql);

It does the job, but the problem of not being able to clearly distinguish your code from the user’s data remains. Here, escaping user data for SQL isn’t as easily solved with the templating system as it is in HTML. Templating SQL code that stores, retrieves, and operates on the very thing your application code depends on, is quite challenging. The method of escaping has been prone to user-space error for many years. Until DBMS developers discovered some better options for abstracting the process in much the same way you attempt to do with an HTML templating system.

Prepared Statements With Parameters

Using parameters in a prepared statement means much the same to your DBMS as an HTML templating system means to your application’s business logic. The purpose of the template is to serve as an abstract idea of what you want rendered in the view. Your application code might want to do various things with the data before it is presented to the user for output. So separating what ends up on the screen, from what’s going on behind the scenes in your code, is important. Equally as important, is the prepared statement that makes it possible to bind parameters to values that can never be confused as code.

$pdo = new pdo("sqlite::memory:");

$sql = "SELECT `username`,`userage` FROM userlist WHERE `userid` = ?";
$stmt = $pdo->prepare($sql);
$stmt->bindParam(array(1, $_POST["uid"], PDO::PARAM_INT));

$stmt->execute();

Here the separation between your code and the user’s supplied data is quite clear to your database. The statement is prepared separate from the data and the parameter is used to bind some value into the statement, then both the SQL code and the data are sent along separate paths. We can’t confuse the user’s input for the SQL code or vice-versa. The same goal you hope to achieve when you want that data placed in your HTML, but don’t want it affect the HTML code and without changing what the user handed you.

Validation vs. Sanitization

This another prevalent assumption that if I only take from the user what I need I can keep my application unaffected and working smoothly. The information a user supplies on the web can be vast. We input everything from our names, addresses, zipcodes, credit card numbers, phone numbers, even to entire documents on the web. There’s a great deal of potential for malicious intent to try and sneak bad data passed our applications to break them. However, there’s also an important need to keep in mind that your application is all about the user. If all your code is doing to make things safer is drive the user more and more annoyed with the process of supplying their input or uploading their information you are only degrading the very people you wrote the code for in the first place.

/* If you're doing this you are causing your users a lot of pain! */

$name = preg_replace("/[^a-z ]+/i", "", $_POST['name']);

Here are some reasons why this is wrong.

• Why can’t my name be Robert Jr. or O’reilly?
• Or how about a hyphenated name like Lee-ann?
• Why would you assume my name can only be presented by the letters A-Z?
• Have you never met a Jérôme or an Aimé or a Noël before?
• If you know Afrikaans you should know some vowel sounds in Afrikaans are represented by an apostrophe.
• Your mangling people’s name! It’s not yours to mangle…
• Mostly, YOU’RE PISSING PEOPLE OFF! STOP!

If the data the user has presented you with, is unacceptable for your application then what you should be doing instead is simply validating that the data is acceptable to you and then use that validation result to determine either (A) the data is acceptable and we can proceed, or (B) the data is unacceptable and we must reject it entirely and notify the user to retry according to our requirements. But under no circumstance should you just change the user’s data without notifying them about it and continue on as if what you have is what they gave you.

However, not everything the user submits necessarily requires validation. As developers, we sometimes like to assume that we know everything and that everything must be validated by us before it should be allowed. This is simply not true. We don’t know everything and we certainly shouldn’t have to be the overseeing party of what is or is not allowed as someone’s first and last name. Or characters the user is allowed to publish to a public forum. This stuff does not break our application code.

In retrospect, there are places where validation is a requirement for the application to function properly. For example, we may need to verify that a user has supplied a valid zip code or postal address on in an order form where they are placing an order for us to ship. If the user enters invalid information there the order can not ship and it presents a problem for the intended users of the application software. We also don’t want users entering one or two letters for their password. This makes the account insecure and opens our application up to attacks. So we may want to validate the user has supplied a password of a specific requirement, like say at least 10 to 20 characters (or perhaps just a lower-bound to prevent people from easily guessing a password). We might want to ensure the user includes at least one upper case character or one special character as well to increase password strength and reduce ease of brute force attacks. However, if the user’s data does not meet these requirements we should be rejecting the data entirely and informing the user of the problem so that they may retry. You wouldn’t just change the user’s password to have it meet your needs and simply carry on. That wouldn’t make any sense! How is the user going to know you changed their password? So you also shouldn’t change the user’s data anywhere else unless you’ve made the user completely aware of what you’re doing to their data so that they have an option to decline or at the very least may chose not to supply their data under such conditions.

$regex1 = "/[a-z0-9]{10,20}/i";
$regex2 = "/[A-Z]/";

if (preg_match($regex, $_POST['password']) && preg_match($regex, $_POST['password']))
{
        /* Password is acceptable */
} else {
    /*
        Password is unacceptable
        Reject and inform the user
    */
}

Also, don’t go out of your way to make it very difficult for the user to meet your requirements. For example, have you ever seen an online order form where you’re asked to enter such information as your phone number or credit card number and prompted not to use dashes or spaces? Why should this be a burden the user is faced with? Here you aren’t really validating anything but the user’s ability to follow instructions. Have you ever heard of a regular expression? Have you ever heard of client-side code that improve the user experience? For example, if you want the data sent to your application in a specific format why not make that a part of your front-end? You can use separate input fields to make it clearer to the user. You can also validate the formatting according to regular expressions on the back-end. But don’t constrain the user experience when you have other options that can still ensure your data validated and keep the user happy at the same time.

You’re a CompSci Major And You Don’t Know It

In my provocative attempt to get the attention of a computer science major student I have had to go to a great lengths to make a subtle point. Unfortunately for me this took up a bit of my time. Fortunately, for the person involved in the discussion I probably got through to them. Although, you can never be too sure. Some people simply refuse to accept that they may be wrong about something out of delusions of grandeur, perhaps self-fixation, or even for reasons of pride. Whatever the case may be it stands to reason that no matter how well you think you’ve learned something you may still be wrong. Medicine is one controversial field where this notion holds true on a constant basis. Physicians and researchers in the medical field find out that what they thought they knew actually isn’t true only many years after their work has been published. The same can be said about the field of computer science.

Allow me to explain how this discussion began. I was speaking with a programmer – let’s call him Foo – on an online forum for PHP. Now, Foo, is new to PHP. He’s trying to optimize his code and has posted some PHP code on the forum looking for advice from more experienced programmers in PHP. So far Foo seems like the typical case where they understand the language syntax and just need assistance making the working code better. Now, another programmer steps in on the forum – let’s call him Baz – and contributes at length with suggestions that seem to me like pathetic micro-optimizations that aren’t really going to help Foo. His suggestions begin with things like use shorter variable names, don’t define keys for arrays, use include and not include_once, and use switch and not if constructs to make your code faster. I start to see Baz is clearly not as experienced at PHP as he’s trying to make himself out to be.

Let’s examine why I think so and why Baz might be very wrong.

Will The Real PHP Please Stand Up

First, Baz is making some non-optimal suggestions that don’t necessarily improve the performance of your PHP code, but might actually break it. For example, suggesting the use of include vs. include_once for performance reasons is not a reason at all. Since both do different things they have different uses. You should not be basing your use of either construct on performance reasons alone. You definitely want to use include_once where your intention is never to re-parse the same file twice; where the order of the calling and called scripts is not clear.

Second, it can not be easily determined that using either include or include_once will hinder performance or improve it. The include_once call itself is merely a wrapper around include that checks the list of already included files. Yes, you’ll be hitting the hashtable, but if you don’t already know it yet PHP hits that hashtable for virtually everything you do in PHP. The hashtable was micro-optimized to be fast for this very reason. Additionally, such a performance hit isn’t even considerable when the need for include_once is apparent. This is like saying to improve my car’s gas mileage I’m going to take out my backseat. Sure, removing any weight from the car means less energy required to move said car, but when you understand the mechanics of the modern motor vehicle and apply just a little bit of common sense you soon realize this is a pretty silly move when you have passengers on board and they are uncomfortable or even injured on the ride because you don’t have a back seat in order to spare $0.01 of gasoline costs on a 15 minute trip. If you’re so concerned with cost you really need to seek out an entirely different solution. Like move away from the horrid design of the combustion engine altogether!

Third, suggestions like a switch is faster than the if construct are completely unfounded and very much baseless over-generalizations. The semantics of the if and switch construct are pretty similar in most high-level languages including PHP. They may differ slightly, but most of it comes down to simple evaluation and jump operations.

When Baz began explaining to me that I’m an idiot that doesn’t know what he’s talking about, I merely shrugged it off as someone who was about to rant on a public forum that there was an obvious performance benefit between switch and if in php. So I moved on. Then when Baz realized I wasn’t responding to his rants he began daring me to disprove him. This is a common false burden of proof fallacy since he is the one posing the argument it really is his burden to prove himself right and not make me do the work to prove him wrong. But, in good nature I laughed and proceeded to prove the poor compsci student wrong. To do so I showed him the branch analysis of an if versus a switch construct using the same 4 expressions to simply check for a true or false and print a single line.

PHP IF Branch Analysis

PHP Switch Branch Analysis

Now, here’s the kicker. If you want to be pedantic about it… The switch will actually cause PHP to generate more opcodes than using the if/elseif construct in this case. This still really says little about performance though. Since they ultimately just lead to a simple branch analysis in the end. All PHP is doing here is evaluating the expression given, to a boolean value of either true or false. If it is true then it executes the opcodes for that branch, if not it moves on to the next branch.

Here’s the code for both scripts.

IF – test1.php

$baz = false; $bar = 0; $foo = ""; $var = true;
if ($baz) {
        echo "We have baz!\n";
}
elseif ($bar) {
        echo "We have bar!\n";
}
elseif ($foo) {
        echo "We have foo!\n";
}
elseif ($var) {
        echo "We have var!\n";
}

Switch – test2.php

$baz = false; $bar = 0; $foo = ""; $var = true;
switch (true) {
        case $baz:
                echo "We have baz!\n";
                break;
        case $bar:
                echo "We have bar!\n";
                break;
        case $foo:
                echo "We have foo!\n";
                break;
        case $var:
                echo "We have var!\n";
                break;
}

So all of a sudden, Baz doesn’t seem to have a leg to stand on and wants to start arguing branch analysis theory with me and vivaciously explaining what little he seemed to remember from his professor in his last ASM class. I could tell he was taking out the book at this point, but rather than be drawn into a pointless debate about things that aren’t helping the status qua, I tried reminding him to get back on topic and help out Foo, instead of trying to convince me he was a smart compsci student. Baz seemed to be infuriated with this and resented getting back on topic. To the point where Baz had now been banned from the discussion and continued to pursue me in private sending me quote after quote from his books.

To indulge the poor fellow I offered some pointers on what it really comes down to when you step back and look at the bigger picture. PHP is not a very efficient way of doing things to begin with. PHP is built on share-nothing architecture. It’s an interpreted language, which means you’re basically recompiling your program from scratch every single time you want to run it. It doesn’t break down to simple x86 ISA in the end and you can’t justify low-level micro-optimizations in PHP since it will ultimately just end up breaking your code in the process of trying to make it faster. The smart PHP developers, the ones who really know PHP inside and out, will always tell you don’t try to outsmart the interpreter, because the interpreter will likely keep outsmarting you.

The Interpreter Is Out Smarting You!

Take people who try to force references into their code as a good example. They think that by using references everywhere they will save on memory and will ultimately make their code faster in ways the interpreter couldn’t. Let us examine the following code snippet to see why this can be a horrible idea and isn’t really offering any benefits worth pursuing.

$array = array(1,2,3,4,5);

foreach ($array as &$value) {
    /* Do some stuff here... */
}

Alright, so this looks great. We didn’t have to use more memory and we can modify the array values in our loop, right? Wonderful! Now we need to just finish this code with one last loop where we’ll print the array elements to the page in a table.

echo "<table><tr>";

foreach ($array as $value) {
    echo "<td>$value</td>";
}

echo "</tr></table>";

The result of our code is now:

1	2	3	4	4

If it isn’t already obvious to you what happened here… You basically just got outsmarted by the interpreter. In trying to outsmart the PHP interpreter you ultimately broke your code and now have an undesired side effect. This is completely expected behavior, by the way. The reason for this is because $value is still a reference. The reference never went away, right? You made it a reference so its use anywhere else in your code still keeps it a reference. And since you decided to use it in the next foreach loop PHP is still assigning the value of each element upon iteration to the $value variable and thus upon the last iteration we’ve now reassigned the value of this variable by reference. Yes, you brokeded it!

Not to worry though. When you’re iterating over an array with foreach PHP is already using the same amount of memory you would use as if you assigned the variables by reference anyway. Why? Because PHP is using copy-on-write. Which means it doesn’t use the memory unless there has been a write operation that has now changed the values of one of the variables. Otherwise the variable is nothing more than an extra refcount for the ZVAL.

Goodbye Baz

Now, Baz eventually gave up and agreed that what he was arguing was not only baseless egocentric banter, but also not very helpful to Foo. I explained to Baz that he should never think that information from a single source amounts to intelligence. It’s just information. Intelligence needs to be gathered from multiple independent sources and corroborated through peer-review. Otherwise, it’s just pointless to say that it’s intelligible when you can’t even get your peers to agree on it.

So the next time you get a compsci student trying to tell you off like he’s the boss, ask him or her an intelligible, objective, question. If they can respond with an intelligent and completely objective answer they might be worth listening to. Otherwise, tell them to go back to school. Computer Science students that have the attitude “I learned it this way and it can never possibly be right any other way” are always going to find it difficult to get work once they step outside of their school and into the real world. Word to the wise: CompSci Geeks DO need glasses!

The Truth

The truth is the world has enough renewable energy sources to power our planet and light it up like a Christmas tree for many — many — centuries to come. The truth is renewable, clean, and natural energy is far more efficient than any of the fossil fuel based energy we use today. In fact, the truth is we are not the technologically advanced civilization we think ourselves to be. We’ve been taught in school, for many decades now, that ancient civilizations were primitive people, using stone and copper tools, hunting and gathering, and eventually we built up to civilizations that cultivated the land and so began the manufacturing age. As though mankind has been on a linear course of progression for thousands of years and it’s just now that we have discovered science and technology and things ancient civilizations could not possibly have fathomed. As if we are thinking up computers and massive global networks for the very first time in mankind’s history. Evolutionists would have you believe this fits perfectly with their theories and politicians lining their pockets with the fat-checks of money-hungry oil conglomeration can do little more than back such ridiculous notions. Not based on any shared beliefs or ideologies with evolutionists, but strictly based on math.

If the world were to discover that oil and coal are not the greatest source of energy there would be wide-spread downfall of international energy corporations and all of their infrastructure. Many interested parties would take on insufferable losses and their business would surely vanish like the ancient civilizations we no longer see today. I imagine, that if the ancients were still around today they might actually be the ones laughing at us and calling us the primitive ones. Why is it so inconceivable that other ancient civilizations might have discovered the immense benefits and substantial gains from building global long-term infrastructures that would harness and support technological advancements? Is it really so inconceivable that ancient civilizations could have had sciences and technologies just as advanced as ours, if not more? Unfortunately, science can not teach us what could be, just what is. Yet, it is through science that I can show you just how much you have been deceived.

The Advent of Electricity

In order to see where we went wrong with energy, we must first begin our trip more than 120 years ago when electricity was just taking form in the heart of the United States. During the late 1800′s America was just beginning to form infrastructure that was quickly changing the world. The rapid development of the telephone and now electric power plants. This was at a time when Edison was just getting started.

However, most of us only know one half of this story. We tend to have a biased opinion about energy problems today, primarily because we have insufficient information to actually make an informed decision about the subject at hand. We are ill-equipped to formulate opinions, because we lack the analysis of the full story.

In the late 19th century a man named Nikola Tesla began bumping heads with Thomas Edison. Tesla, worked for Edison at the time. Edison wanted to use DC – Direct Current – which was inefficient and proved difficult to supply due to its limitations in transport. Tesla wanted to use AC – Alternating Current – which proved superior to Edison’s Direct Current. A direct current travels in one direction whereas an alternating current can travel in many directions at the same time. However, Tesla was hardly recognized for his inventions and radical innovations. He was considered outcast from the science community and ultimately went into seclusion. It was not until decades after his death that people began to truly realize the genius that was Nikola Tesla.

Tesla, none-the-less, was a visionary. A mind well beyond his time. He tried demonstrating his device (the Tesla Coil) for using Alternating Current at the World Fair in 1893 — World’s Columbian Exposition in Chicago.

Within the room was suspended two hard-rubber plates covered with tin foil. These were about fifteen feet apart, and served as terminals of the wires leading from the transformers. When the current was turned on, the lamps or tubes, which had no wires connected to them, but lay on a table between the suspended plates, or which might be held in the hand in almost any part of the room, were made luminous. These were the same experiments and the same apparatus shown by Tesla in London about two years previous, “where they produced so much wonder and astonishment”.

What a remarkable idea, right? Wireless electricity developed in the 19th century? We’ve only just began to experience wireless technology in mainstream society not more than at the turn of the 20th century. Yet, here is a man far removed from our times that already thought of this idea. In fact, Tesla built a tower, called the Tesla Tower (also referred to as Wardenclyffe Tower) in Shoreham, Long Island, New York for wireless telecommunications that operated between 1901-1917. Eventually, his investors, J.P. Morgan, cut off his funding because Tesla wanted to use this tower for allowing people to transmit telephony and broadcast wirelessly and for free. Of course his investors could not make a profit this way so they pulled out and the tower was never complete. Imagine, wireless, limitless, global communication, that was free! Does the Internet start to sound like less and less of a new revolutionary idea, now? Perhaps you might still not think so, but what will surprise you next is that this wasn’t even the first time someone has proposed, or even built such a technology!

Back To Ancient Egypt

In order to understand even further, where we went wrong with today’s so-called “energy crisis” we must travel even farther back in time to the time of Ancient Egypt. The time of the Pharaohs. A time when civilization had clearly left its mark on the face of the world. Man-made structures. Tall, magnificent structures left behind by an ancient civilization that is no longer around today. Heritage, culture, language, artifacts, and even a tremendous wealth that were all left behind for us to discover. From these remnants what scientific evidence have we actually discovered about this civilization?

You may have been led to believe that the Pyramids of Giza were used as tombs for Kings. If this is what you were told then you were lied to! Albeit, it’s quite possible that the people who lied to you were themselves lied to or just merely misled. It wasn’t an intentional sinister lie. It was one of misunderstanding and ignorance. Archaeologists have been hard pressed to explain the many mysterious of the Pyramids of Giza, because they were among the largest and most mysterious of all the Pyramids we have discovered today. Despite the mostly mythical notions about these Pyramids in Giza, Egypt being used for  the tombs of Kings, the truth is they were the power plants of the ancient world. Yes, Pyramids, were power plants of Ancient Egypt. How? Science tells us the story we were too stupid to understand when we first started exploring them.

These are hieroglyphics from a temple in Egypt depicting that Pharaohs used light-bulbs long before we ever thought electricity had been discovered. Furthering the evidence for this claim is that if the ancients did not use electric light bulbs (depicted by the Pharaoh in this hieroglyph holding a bulb connected to a wire which is powering the luminous bulb from the Sun God Ra) they would have had to use flaming torches to see inside the corridors of the Pyramids. Torches would have left a residue of soot inside the chambers and corridors of the Pyramids. However, there is no evidence of any soot. The ancients did not see in the darkest bowels of these structures by way of fire as we once had believed.

It seems that the ancient civilization we deemed inferior to our own in technology and scientific advancements were so superior that ironically we failed to realize by just how much. Not only is there scientific evidence of technology apparent in the inscriptions left-behind from this civilization, but there is proof in every aspect of their work that we have only recently begun to unfold within the last few decades. For one thing, the mummies discovered in Egypt were all found in burial places such as the Valley of the Kings, which is a site far far removed from the Pyramids of Giza, or in other smaller pyramids and tombs that are not the great pyramids of Giza. There was not a single mummy found in these Pyramids. In fact, there isn’t even a single Hieroglyphic marking inside the Pyramids. Most tombs, temples, and burial places exhibited explicit inscriptions about their purposes; clearly marking that they were the resting places of Kings or Pharaohs. The Pyramids of Giza, however, did not. They remain a deep mystery – to this very day. Some of their secrets are yet to be discovered.

Think of it this way. If you were building a nuclear reactor, would you decorate it with art work and describe its uses and purposes openly? Surely, you would not. Today’s nuclear reactors and power plants are tightly secured structures; their secrets close-guarded by those only granted clearances into its deepest borders. Like-wise the Pyramids were only unraveled through close observation and careful study. Early Archaeologists only depicted these structures as the tombs of Kings because when they were questioned about their purposes and significance to the civilization they could come up with no other logical explanation. So they simply tried to deduce what might have happened to the mummies by hypothesizing that the sarcophagi and/or their mummies had been stolen by grave robbers along with any riches they may have found within. However, there is no evidence of there having been any riches in these structures. Would you store money in a power plant? There is no evidence of any sarcophagus lingering anywhere about in the corridors or chambers inside of the Pyramids. What we found instead in the Great Pyramid of Giza was what looked like it might have been a place for a sarcophagus, but turns out to have had a much different purpose.

Buried deep beneath the Great Pyramid of Giza sits a chamber with a single granite compartment showing clear evidence that it had been carved from a single slab of a special kind of granite. Granite is naturally radioactive. Certain types of granite have proven to have up to 10 PPM (parts per million) uranium. The zig-zag style corridors inside of the pyramids indicate passage way to aquifers leading from the Nile river beds to the underground chamber beneath the Pyramids vast structure. The water was used as a conductor in the production of electricity. Though the Pharaohs were not manufacturing electricity the way we are used to today. They were harnessing natural electricity. In Physics, we refer to this as subtle energy. It’s the use of implosion energy, not today’s more common explosion energy that set this civilization apart from ours by centuries and centuries of being in the dark ages. The Pyramids were coated with limestone from the outside. Limestone is a poor conductor of electricity. It was used as an insulator (in its pure calcium form). The granite, being a good conductor of electricity was used throughout. The top of the Pyramid was fitted with a huge block of gold as a cap. This is no longer present today, but at the time before it had been removed – or stolen – it was believed to have been used to conduct electromagnetic energy from the sun (the ionosphere). Gold is a perfect conductor of electricity and this would have been the equivalent of today’s solar-power cells used in so-called green-energy. The Pyramids, however, did not just produce or harness the suns energy or the planets natural energy – they had an even greater functional purpose. They were engineering feats of the ancients of our world to build infrastructure that would supply all of Egypt with wireless, safe, and limitless energy. They were crafted with the utmost precision.

Imagine, not only could you use these structures to power light bulbs and other tools, but you could power the people. The human body is the perfect conductor of electricity and can be used to safely pass electric energy current through the skin without harming the human being. Portable, efficient, safe, and best of all renewable energy! Is this not the mark of a genius civilization? A technologically advanced civilization? Yet, today we pride ourselves in having built microchips that produce immense heat, cost untold amounts of electricity – a result of harmful fossil fuels, and plead to be advanced.

The Pyramids are now theorized to be the greatest proof that ancient civilizations were indeed in position of advanced technology that even today we have not managed to acquire. In fact, with all of our technology today we still can not reproduce these magnificent structures that lay and remain in their place for more than 5,000 years!

The Great Pyramid of Giza is made up of more than 2.3 Million stones. Each stone weighs in at around 2 or 3 metric tons. That means this structure required people to cut, move, and assemble more than 6 or 7 Million tons of stone. By comparison, each of the World Trade Center Towers of New York City weighed around 500,000 tons. That’s an order of magnitude less than the Great Pyramid of Giza. Additionally, we know where the stone came from because of its mineral compositions having been tested. They were hauled from mountainous regions hundreds of kilometers to the south of Giza. Some Archaeologists might have you believe that these stones were moved using ropes, and piled on top of logs and raised with simple pulleys. Dragging two metric tons across the desert over a hundred miles? Sure, these men and woman all had Ph.Ds from fancy Universities! Lets give them credit for their imagination if nothing else, but in what science can you convince the world this explanation can be scientifically proven? Where did we find millions of logs of timber in the middle of the dessert? Further more, how many man hours do you calculate it would take hard laborers to move all these stones? It would have taken centuries at their populations. Beyond that there is evidence of pure craftsmanship everywhere in the Pyramids. The stones are cut to a fine edge with what could not have possibly been accomplished using mere stone and copper tools of simpletons. The stones are assembled with such precision that it defies all plausible rate of error, beyond that of which our modern-day engineers can even fathom. This is simply outstanding evidence that we haven’t even bothered to consider the possibility that perhaps it is us who are the inferior civilization. Perhaps we are only assuming the ancients were living in the dark ages, because their knowledge had been lost and it is our ancestors that were the ones thrown into the dark ages only of late. Now that we are merely just emerging from these dark ages it can not be so difficult to believe that previous ancient civilizations may themselves have once been at the height of technology and scientific advancement. Perhaps even so much so that they surpassed anything we have done to this very day by centuries.

You Are Not Dependent On Fossil Fuels

The lesson to learn from all of this is that fossil fuels are not the only source of suitable energy that can power our planet. Not only did scientists find proof of this technology among the Ancient Egyptians, but even the Mayans, the Chinese, and ruins discovered in European countries demonstrate the existence of Pyramid structures all around the world that may very well have been used for the same purposes. Tools that would compare to today’s only recently sought technological advancements like the quartz crystal, power tools that can cut through hardened materials like granite, and much much more.

However, as long as there are greedy people in the world who would have an interest in controlling the worlds energy markets and controlling you to continue buying from them what seems like a product that can be depleted and run out – there will be such gimmicks to try and keep you in the dark about how much we can do without our horrible idea of energy today. The harm and the costs that it has burdened us with is beyond unacceptable.

It’s no new notion that we should turn to alternative, clean, renewable energy sources. We’ve been arguing and debating about this for decades, perhaps even centuries. Take a look at what the smart corporations are doing today about the energy problem. Google has invested a great deal of money in finding clean, renewable energy sources. They have even recently started a subsidiary – Google Energy – to fund this initiative and perhaps even pave their way into reducing the worlds energy costs through clean renewable energy. Google actually finds locations for its data centers in Asian countries were a steady flowing water supply, such as a river, or an ocean is nearby. It engineers and powers its resources based on ancient technology (building damns for power). Natural energy is all-around us. It flows freely throughout the earth in ways we are only beginning to discover. Tesla discovered this in the late 19th and early 20th century when he demonstrated how his device could power a light bulb without wires by allowing the alternating current of thousands of volts to flow through a human body and into the bulb, without harm to the person. Today, trying to touch a high-voltage wire of just a few thousand volts would surely kill you. Tesla’s Tower was powered by aquifers very similar to the ones the Ancient Egyptians used in their Pyramid. Today an unfortunate natural disaster in Japan nearly caused a nuclear melt-down and put millions of innocent people in harms way. There is no need to even mention the war, death, and destruction crude oil has caused us in the last few decades alone. Why are we bringing harm upon this world in order to help its people? If energy really is supposed to help us produce technology that helps and not harms the civilizations of this world then why doesn’t it?

I urge all of you to ask more questions and accept less things at face value. I encourage you to explore the true nature of science and technology as you are being taught in your schools, your homes, your places of business. Where does the world’s interest truly lie and who is trying to simply cover your eyes in order to fill their own pockets? If we were not really taught the truth in history class — how wrong could our science classes have been?

For example, websites like reddit, mozilla, imgur, and wikipedia‘s English site have chosen to completely blackout from the public. While wikipedia is easier to get around than others sites like reddit, mozilla, and imgur are pretty much going to leave you empty handed for the day. Google simply blacked out their logo today and provide a direct link on their front page. If you’re visiting Google for the first time today you might have been redirected to the announcement page to contact your local senator. Beyond that they don’t seem to nag you like other sites are doing. Sites like WordPress, php.net, and Google took an elegant, but effective approach to the blackout. They demonstrated their anti-SOPA while still keeping their visitors happy. Sites like Bing and Yahoo, however, don’t seem to have participated in today’s blackout in any noticeable way.

Internet Blackout - SOPA

Read more about the SOPA Strike here.

Databases Are Overkill

For those who start off building small web-based applications or even trying to put together a tiny CMS (Content Management Systems) they sometimes fall victim to the illusion that having a very small amount data would mean that building a database for this data would be overkill. This is simply not true anymore. Today databases are easier than ever to build, grow, and manage. With lite-weight solutions like SQLite you actually improve on performance with small amounts of data and make it easier to manage. SQLite is actually a small foot-print library written in C that implements an embedded DBMS. It’s only a few hundred KB in size and implements most of the SQL standard. You can use it to store databases in memory or on disk and still get the full benefits that relational databases offer with a minimalistic foot-print and without compromising on performance for small data sets. It’s adopted by PHP, Python, Perl, Ruby and even Javascript as well as many other languages. So there really is no excuse to avoid using a database when the solution is widely available in so many popular platforms and especially in web development.

Databases Are Slow

This could not be farther from the truth. A relational database can maintain indexing for records across different tables. This means rather than looking through the entirety of the data set and then trying to expose some underlying structure in order to find a particular set of data the relational database takes advantage of composing structures as you build your data sets. These structures make things like fetching a record with a primary key much much faster than you would get by using a flat-file solution.

Lets examine the alternatives. Even if you had a very small amount of data – say just a few hundred lines of text. Even if the data structure was overly simplistic – we’ll assume each line represents what would be a single row in a database table. Even if the data will only ever be maintained by a single developer – you. You are still overlooking so many problems that are not easily solved by using a flat-file to maintain this data. First lets consider the race-condition. You have a script that opens a specific file on the server and appends a new line each time a record is added. The script can also open the file for reading and retrieve the entire contents of the file into memory. The script can then do any necessary sorting and filtering to return the required data sets to the user. The most apparent problem with this approach is the race condition. It is entirely plausible that two requests could be made simultaneously to the same script – one to open the file for writing and append a record and one to open the file for reading and retrieve the data. If the data is stored into memory before the line is appended the result is stale and potentially corrupt. If the new data is appended before the read, no problem. However, what happens when you want to delete a record. Now the problem is three-fold. If three individual requests all come in at the same time – one to read, one to write, and one to delete a record – it is now likely the case that your entire data structure has been corrupted. Remember that HTTP is built on a request-response model and no two requests are treated as if they are tied to any previous requests. So there’s no central point of control over your script’s ability to manage which process can access the data and to what extent.

In a DBMS, on the other hand, the control is transferred away from the script and to the central management system of the database. The DBMS then gets to decide how requests will be served and the order of treating the data. This creates more dependable data that has a far lesser chance of corruptibility. Now, it’s entirely possible that you may not be concerned the integrity of your data for a small application, but then you might as well not waste your time building it.

I’ll Use A File Now And Learn To Use A Database Later

If you’ve said this phrase it’s already too late. It doesn’t take a lot of time to get started with a database in the first place. If you’re using languages like PHP, Python, Perl, or Ruby you probably already have the necessary libraries installed on your system to work with a database. These libraries and drives are usually packaged with these software stacks as standard. It’s actually uncommon to not have some DBMS solution already available in most of these environments. So why would go out of your way to reinvent the wheel when the solution is already at your fingertips? Not only that, but it takes very little time to set these DBMS solutions up and get them to run smoothly on virtually any platform. You will probably spend more time trying to write a script that stores, retrieves, sorts, filters, locks, and validates data using a flat file then you would installing the DBMS and getting a simple schema started.

If you’re using PHP interfacing with a database has become easier than ever. It only requires a couple of lines of code to open a database connection to virtually any database you have a PDO driver installed and loaded for in PHP. So whether you’re using SQLite, MySQL, PgSQL, etc… you shouldn’t need to spend a lot of time learning how to interface with each of these databases if you simply stick with the PDO extension. You use the same functions regardless of the database. This is opposed to having to learn the individual database-specific extensions in PHP to interface with each of those databases. Not to mention PDO supports many of the popular new database features such as prepared statements and is a lot easier to learn and use than extensions like MySQLi.

PHP and Databases

Being a PHP developer, I also take notice of many PHP developers that tend to have the misconception that when they start using a database (usually it’s the case that their first database is MySQL) they should start by learning the old mysql extension in PHP. This is simply not true. Some of the underlying reasons this is widespread, however, is mainly due to the fact that the old mysql extension has been around for quite a long time in PHP and it’s fairly common to see a lot of PHP code demonstrating the use of a database in PHP using this extension. It’s also become familiar to a lot of old PHP developers and is bound to be present in their older applications. However, the use of the old mysql extension is highly discouraged for new development. It’s an old extension that’s no longer well-maintained and has been planned for deprecation for years. There’s no guarantee that if a new bug creeps up that someone will go back and fix it. This leaves your application vulnerable and exposed. If the code base gets large enough this might leave developers scrambling for migration path. Additionally, the extension does not support prepared statements or parameterized queries. This makes things like making sure you properly escape user data to avoid SQL injection, prone to error. The extension lacks in many areas that are not conducive to future development. Learning the old mysql extension first before you learn the new improved mysql extension in PHP or before you learn PDO will gain you nothing. In fact, it will cause you to have to unlearn some of the very poor design of the old extension and its implementation details in order to become more accustomed to the newer extensions.

Some developers also complain that PDO seems too complicated or more difficult to use than the old mysql extension. This might come from the lack of understanding as to what PDO even is or how its used. Since PDO can only be used with the newer OOP features of PHP (you have to use objects and methods instead of procedural-style functions) it makes it seem unapproachable or even scary to developers who aren’t used to OOP in PHP. There is also the idea that PDO has a lot more features due to its vendor-agnosticism and the fact that requires further configurations such as installing and loading the individual drivers needed for interfacing with your specific database (where the drivers aren’t already packaged or loaded). I can understand the intimidation, but most of this has been alleviated with new versions of PHP coming pre-packaged and loaded with most of the popular drivers and the documentation offering up examples that are now easier to follow and get started with. Most of the intimidation is actually coming from having to unlearn old habits that older extensions like the old mysql extension once taught.

Once you get past the initial intimidation phase and actually get started with a PDO and with a database you’ll find that it doesn’t take nearly as much time as you’d think to get up and running. Most of the reservations people have are 90% of what’s holding them back. Not that the investment to get started is actually that significant. Beyond that you’ll find that learning to build on data normalization not only make development easier, but makes your users happier. When you can organize and maintain data that’s more clearly structured and accessible you can serve your users more effectively and efficiently. That will keep users coming back and eventually help you grow your application!

When you visit most websites that require you to log in in order to access user-specific content, which requires some form of authentication, you are bound to stumble across the “remember me” feature during the authentication (or sign-in) process. This is usually seen as a checkbox along with the username/password fields next to the sign-in button on the authentication page.

If you check this box, to remain signed into the service or website you are authenticating to, then what happens is the server sends you a cookie back in the response header of the HTTP request, that provides some information which allows the server to re-authenticate you, if needed, upon future requests where your existing user session has expired. I’ve seen a lot of developers implement this feature incorrectly with PHP so I’m going to try to explain in detail how you should be doing it and what you should try to avoid. Here’s a hint for a common implementation mistake; this feature has nothing to do with your existing session. It should be an extending component of your authentication mechanism, but it should definitely not rely on the existing session since it’s quite possible this session can expire or be lost at any time and without warning. Here’s a more detailed explanation of how it should work.

The Sign-In Process

First, your users have to authenticate in order to gain access to a server-based session. If you don’t already know how to do this with PHP it can be as simple as validating that the user submitted their username/password credentials correctly in the HTTP request by matching them against the credentials stored for the user in your database, or however else you implement this. If the authentication is successful you would normally have started a session in PHP by using the session handler. A simple call to session_start() will start the session handler, check the request headers for a valid session ID (or SID) according the session_name directive in PHP, open the existing session file on the server for reading or create a new one if one does not already exist for the supplied SID (the session file is also locked), unserialize the session data and populate it to the $_SESSION superglobal (if any session data exists in the session), and once script execution ends or the session handler is called to close the session any data left in $_SESSION is serialized and written back to the session file and the exclusive lock on the session file is released. It is then your job to store some data in this session file to indicate to your subsequent PHP code that this user has authenticated successfully and any future requests using this existing session will not require the user to re-authenticate by entering their username/password, for example. This could be as simple as adding the user-id to the $_SESSION superglobal. You could also use some boolean value for authentication like $_SESSION['authenticated'] = TRUE; but you still need to know the user ID from database in most cases so it makes sense to store it there.

The Problem

This all works great until the garbage collector comes around and eventually removes those session files on the server. The reason is because session data is meant to be temporary. It should be something you expect to lose at some point and you should not worry about losing it at any time since your real persistence layer should be your database or some other permanent form of storage. PHP’s session handler comes equipped with a number of directives to control how the sessions will work. I’ll explain some of the important directives for the session handler that you should be concerned with in this situation.

• session.gc_divisor
• The gc_divisor is the divisor used by the Garbage Collector along with the gc_probability as a probability factor to determine how often the garbage collector should run. This number should usually be set to 100.
• session.gc_probability
• The gc_probability is used along with the divisor to get the probability for running the garbage collector each time a request is run through PHP. This number is usually set to 1 and should be increased or decreased accordingly depending on the load of the server, number of users, and number of requests. The formula is basically gc_probability over gc_divisor equals the chance the garbage collector will be invoked. So if these numbers are set to 1 and 100, respectively, the chances the garbage collector will come around is 1 out of every 100 requests. The reason you may want the garbage collector to run more often is if you have a large number of session files on the server and you don’t want to slow PHP or the server down by having to delete huge amounts of these files at a time. The higher probability may help in this situation. Adversely, if you get too many requests too fast the GC (Garbage Collector) may be slowing you down and the probability may need to be lowered.
• session.gc_maxlifetime
• The maxlifetime directive is telling the garbage collector how long (in number of whole seconds) to allow a session file to remain in the session directory before it is considered garbage and removed by the garbage collector. This is usually determined by checking the file’s atime or mtime (depending on your file system this information may not always be available and may be ignored completely thus eliminating the garbage collector) against the system’s current time. If the difference exceeds the number of seconds specified in this directive when the garbage collector comes around, then it is considered flagged for deletion and should be removed by the garbage collector.
• session.save_path
• This is the path PHP will use to store session data when the session handler is set to save sessions using files, which is the default behavior. This path needs to be writeable by PHP and there are some things you should consider when this path is shared by multiple scripts using different session directives. For example, if you have two different scripts each specifying a different session.gc_maxlifetime directive using the same session.save_path the garbage collector uses the lowest value to determine when to mark files for deletion. Also since PHP’s session handler does not protect against things like session collisions you should not increase the session.gc_maxlifetime directive beyond 20 or 30 minutes (depending on the server load). Otherwise you increase your chances of creating a session collision substantially if the traffic to the server is high enough. It’s not entirely unreasonable for two session requests to be generated at the same exact microsecond and chances are this could likely result in two sessions assigned the same session ID and now one of two users may be logged into the other user’s account through no fault of their own.
• session.cookie_lifetime
• This should not be confused with directives affecting your server’s session file. The session cookie is sent to the user. It contains the session name and session id of their session. The session name is specified by the session.name directive. The session id is the name of the session file on your server (but does not include any path), which is just a random hash generated by PHP using the session.hash_function and session.hash_bits_per_character directives. The default hash function is normally 0, which might be MD5 – a 128-bit hash. You can get a list of all the supported hashing algorithms on your system using the hash_algos() function in PHP. This should return an array of all the available algorithms and their corresponding index. This is the number you specify for the directive. The session.hash_bits_per_character may be set to either 4, 5, or 6 and that just tells PHP how many bits from the binary data produced by the hashing algorithm to store in each alphanumeric-character sent to the session handler for producing the session ID. The default is usually 4. So for a 128-bit MD5 sum this would result in a session ID of 32 alpha-numeric characters.

So once your users’ sessions have expired the PHP code that depends on the session for validating authentication will fail and the user will be forced to log in again or re-authenticate. This can be a little frustrating for the user depending on how often they use the service and how long it’s been between requests. I’ve seen a lot of developers in PHP make the very poor choice of raising the session.gc_maxlifetime directive to something ridiculous like 24 hours or even 30 days to avoid having the user re-authenticate and keeping them logged in. This is just plain wrong and can lead to horrible compromises of your system. This is NOT the way to implement a remember-me feature for your users. For one, if you allow the GC (garbage collector) to keep files around for hours or even days at a time you have exponentially increased the probability that PHP can generate a new session id for one of your users that already exists. Remember, PHP’s session handler does not protect against session collisions. It has no idea if a file already exists using this session id in your session.save_path. It doesn’t bother to check. It just generates a random hash and if the file already exists it’s overwritten. By lowering the max_lifetime of your sessions you force the garbage collector to remove older data and narrow down the possibility of a collision.

Think of it this way. If your server is getting around 100K hits per hour and there are about 1K unique visitors per hour and they each generate an average of four or five session hits per hour the probability that one of them will make a request to an expired session is fairly high depending on the frequency between hits. This is all assuming your session’s max_lifetime directive is set to 20 or 30 minutes (at the highest). If you increased this directive to 30 days the chances that any of them will hit an expired session within the hour (causing the session handler to generate a new session) has become 0. What this means is PHP is generating a new session file every time you call session_start() unless this user has sent a valid SID in their request header. So if a user has cookies turned off or their browser has expired the cookie sent by your session handler the server is constantly generating new sessions for all of these users and it’s not deleting any old sessions until they’ve reached this 30 day period. It’s possible to fill up your server with hundreds of thousands of new session files each day in this manner. This means in a month you could have millions of session files and the possibility of a collision is far more apparent.

The next mistake I see some PHP developers making down this path is trying to move the session handling away from the session handler and into their database and user-land code. This is also a pretty bad idea. For one, your database is already your biggest bottleneck 9 out of 10 times in virtually any application. Second, if you aren’t using the session handler you’re wasting a lot of time writing code, in user-space, that’s already been written for you. You are only increasing the chances that you will have more buggy code without thorough testing or ultimately just wasting more time reinventing the wheel. PHP’s session handler is a lot more powerful than most people think. You can even define your own custom session handler if you wanted, but normally the default session handler works just fine even on large scale systems in all the cases I have I worked on. I’ve had no real trouble as of yet getting sessions to work even on large clusters just using PHP’s default session handler. If you’re going to be adding some memcache or reddis on the side for whatever purposes your application requires, that’s fine, but the session handler works great for handling sessions. The additional functionality you may choose to include through your database or other cache are just added bonuses, which is what the remember-me feature is all about. It should have nothing to do with your session at all. It is a completely separate functionality that may or may not work along-side of your existing user session code, but isn’t a replacement or extension of the session itself.

The Solution

What you do want for the remember-me feature is a completely random value that’s stored in your database along with the user’s credentials. So, for example, you probably already have a user table in your database that stores the username and password for each user. Simply add a column to that table for the remember-me feature. You normally want this to be a VARCHAR or TEXT column since it will store an arbitrary amount of data from a binary source of entropy. You likely also want to set a unique constraint on this field so that your dbms will prohibit you from storing any two rows with the same exact value. This will prevent users from inadvertently being signed into each others accounts in case of a collision. Next you want to produce a completely random string for this field. It should have nothing to do with the existing data about this user. I see a lot of people making the mistake of trying to hash the username/password along with a unix timestamp, for example. This is just wrong and you don’t ever want to do this. The reason is because it’s easier to try and reverse engineer the algorithm you used to build the hash than it is to try and bruteforce my way through if I’m trying to compromise your system. If I figure out how you formulate the hash I can easily reproduce that hash and attempt to gain unlawful access to a system. This might not sound very plausible to the average person, but for a skilled Infosec guru this is child’s play. A hash using time() in PHP is not cryptographically secure since if I make enough requests to your server to generate multiple hashes I can ultimately figure out how to reverse engineer that hash. There’s a lot of trial and error involved, but it’s still easier than attempting a brute force of 2 to the power of 128 per user. The unix timestamp makes your algorithm time-sensitive and now I’ve revealed a reversible factor in the engineering process. However, data from /dev/urandom, for example is not time-sensitive in any meaningful way to an engineer. I can’t tell you what /dev/urandom will have produced 10 seconds ago any more than I can tell you what it will produce now or in the future. But I can, however, tell you exactly what time() will have produced 10 seconds ago as well as now and in the future.

So instead to produce this value we can use the following code

/*
 *  Works on both Windows and UNIX/linux platforms.
*/
$key = bin2hex(mcrypt_create_iv(100,MCRYPT_DEV_URANDOM));

// Or use base64_encode() for transport instead
$key = base64_encode(mcrypt_create_iv(100,MCRYPT_DEV_URANDOM)); // Notice the key is not hashed

/*
 *  Will only work on UNIX/linux platforms
*/
$fp = fopen('/dev/urandom', 'rb');
/*
   We use the 'b' flag for binary so that PHP won't attempt
   to translate any of data such as the line break characters
   on Windows/*nix platforms.
*/
$key = bin2hex(fread($fp, 100)); // Notice the key is not hashed

// Or use base64_encode() for transport instead
$key = base64_encode(fread($fp, 100)); // Notice the key is not hashed

fclose($fp);

On a linux system you can also get the data from /dev/urandom directly using fopen(), for example, and produce the value from the entropy source that way. The above example, however, will work on both Windows and Linux as of PHP 5.3 and produces completely random data. Notice, I’m not hashing the value of the data at all. I’m simply encoding the binary data into a human-readable hexadecimal representation since it will need to be transported over HTTP (it must be URL-encoded). You can leave it as is and just URL-encode the binary data before transport, but that will likely take up a lot more space and you usually want to keep your HTTP response headers fairly small. I chose 100 characters as the payload size and you can either increase or decrease this accordingly to suit your needs, but you usually want to keep at no less than 100 characters in most cases. This extrapolates the possibility for a bruteforce attempt, exponentially so the higher the better.

Next you want to store this value in your database along with the user’s row in your user table whenever the user re-authenticates themselves. So when ever they type in their username/password to sign in and have the “remember-me” box checked upon sign-in you would use this code to generate the random string and store in the database for that user. You then want to send it to them as a cookie during that request.

// Be sure to store the $key value in your database
setcookie("rememberme", $key, time()+3600*24*30); // Set the cookie to expire after 30 days

Now, in the authentication mechanism for your application wherever you check the session to verify the user is still logged in you would ad a subsequent check for this cookie in the event the user’s session does not exist. If the cookie value matches the key stored in your database for that user then you know they’re still authenticated since they chose to remain signed in for X number of days. You then simply regenerate a new session for that user and proceed as normal.

session_start(); // Start the session handler
if (!empty($_SESSION['userid']) && is_valid_userid($_SESSION['userid']) {
    // This means both the session file exists and contains a valid userid in the database.
    // So the user is authenticated and we can proceed as normal.
    /*
       Handle authenticated procedures here...
    */
}
else {
    // This means the session file doesn't exist, is empty, or there is no valid userid
    // This is where we will check for a 'rememberme' cookie if one exists
    if (!empty($_COOKIE['remember']) && is_valid_rememberme_cookie($_COOKIE['rememberme']) {
        // The user has a valid rememberme cookie and the token matches a user in the database
        /* The session is already started so just generate the session data accordingly as if the user has already authenticated */
        $_SESSION['userid'] = get_userid_from_database($_COOKIE['rememberme']); // You need to implement this functionality yourself
    }
    else {
        // User is not authenticated and has no remember me cookie proceed to redirecting the user to the login page.
        header('Location: http://www.example.com/login');
    }
}

Notice we don’t add any username in the cookie since we already have a unique constraint in the database to prevent duplicate values of the same field in the table. We always know the value is going to be unique and whatever row it matches is the user it belongs to. Of course, since this cookie is regenerated each time the user re-authenticates it helps avoid the problem of the user being remembered on multiple machines. If they chose to be remembered from one machine and then re-authenticate on another machine also chosing to be remembered a new cookie is generated, rendering the old one ineffective.

It’s important to note you should have a mechanism in place to either recover from the problem of the value being rejected by the database as a duplicate or notify the user of unrecoverable fatal error due to this problem. The chances of this happening, however, are pretty slim given the size of the random string and the entropy sources involved. Set it high enough and even with a huge database of users it’s fairly low-risk. Given that I can produce over 100K unique keys using /dev/urandom on a dual-core machine in less than 600 milliseconds and consistently produce unique random data over the course of a few days for millions of rows in the database this approach is quite reasonable even for large systems. Here’s how I tested this method over the course of about 7 days for ten million users and came up with only 1 collision out of every one hundred billion attempts. That’s an average of 0.000000001% chance of collision even on high load systems according to my own tests, and keep in mind you can even decrease these chances with better sources of entropy and higher payloads depending on your needs.

ini_set('memory_limit', -1);
$start = microtime(true);
for($i = 0; $i < 100000; $i ++) {
	$key[] = bin2hex(mcrypt_create_iv(100,MCRYPT_DEV_URANDOM));
}
$end = microtime(true);
$time = sprintf('%0.03f ms',($end - $start)*1000);
$c = count($key);
$u = count(array_unique($key));
$n = $c - $u;
echo "Generated $c keys in $time. There were $n/$c repeat keys.
";
echo "

Sample Keys:
\n";
for ($i = 0; $i < 10; $i ++) echo "{$key[$i]}
\n";

I ran similar code an arbitrary number of times (between 1-10 times per minute) at a frequency of one minute for a period of 1 week on a database of a few million users (all random data used for testing) and came up with just one collision. You can easily reduce even this number by orders of magnitudes with just a little bit of work. The fact that a collision can happen doesn’t say anything about the effectiveness of the approach since it’s prevented from becoming a security risk with the unique constraint in your database. The aptitude of the approach, however, is very effective since it doesn’t rely on the session at all and works well alongside of your existing session-based mechanisms, but more importantly it is a completely random token that can’t be reverse engineered (not a hash of some deterministic data).

Synopsis

So in summation what you don’t want to do when implementing this remember-me functionality in your own applications is to make it a feature of the session itself. That’s a big no-no since it requires relying on the session lifetime, at least in PHP, and with the default PHP session handler we explored how risky this is. You do want to implement it as a completely separate feature and not rely on any existing data or hash of the user data to generate the random key. You don’t want to expose any sensitive user information either so don’t send the userid, username, or password in the cookie just like your session cookie does not send any of this information. Instead everything is kept on the server and away from prying eyes. You don’t want any attackers to be able to reverse engineer how the key is generated (just as with the session) so it’s completely random and not a hashed value of some data that can be reverse engineered.

I’m sure many of you have seen this statue before, perhaps not the very same one in the picture, but possibly similar statues around the world. This one is located in New York City.

Statue of Atlas in NYC

This particular statue is the Titan Atlas (a God from ancient Greek Mythology) who was supposedly burdened with carrying the weight of the world – or the weight of the heavens – on his shoulders as a punishment from Zeus. Whether it was the weight of the world or something else is unclear, but most people seem to follow this same observation. In general it’s nothing more than a myth, but the lesson history teaches us is that it constantly likes to repeat itself. Clearly, no one can bear the entire weight of the world on their shoulders just like no one computer can either. If you are running SaaS (or Software as a Service) you are online 24/7 and so is your service. The problem is there are over two-billion users online (or with Internet access) today. What happens when too many of those users all start using your service at once?

What Is Load Balancing

The idea behind load balancing is that a single machine can only handle so much work at one time and you can only go vertical for so high. Notice that even in large cities you can only build so high before you have to start building out. Since on the Internet virtually anyone can be using your server at any time you run the risk of overloading without warning. If too many users all send requests to your server too quickly, the server will reach a point where the load is higher than its capacity and eventually crash. This particular vulnerability of typical client-server relationships on a network is exploited by what is commonly referred to as a DDoS attack or a Distributed Denial of Service attack. Basically, a number of clients (sometimes a bot-net controlled by one or more users) will attempt to send a lot of requests to a server or number of servers very fast in order to overload the server and prevent its intended users from being able to access the service. Sometimes this is done just to destabilize the service running on the server or for other malicious intents. There are ways to mitigate DoS attacks with firewall software/hardware or through other means depending on the service, but not all DoS attacks are malicious or even intentional in nature. Google, for example, experienced what was at first glance considered a DoS attack on its search service during one afternoon on June 25^th of 2009. This actually wasn’t a malicious user or users at all. It was the world receiving the tragic breaking news of the death of Michael Jackson. Literally, millions and millions of users from all around the world flooded Google Search all at once with the same search phrase “Michael Jackson”. Google had never seen such a tremendous amount of traffic coming in all-at-once on a single search query, before, so their first thought was “ohnoes, we’re getting DdoSed!”

Scaling Out - SaaS

Why Do I Need It

The fact remains that any number of users can suddenly surge the number of requests coming in to your servers at any given time and whether that is malicious or not is unimportant. What is important is that you are better prepared to handle such situations so that your service will suffer as little downtime and degradation as possible. So load balancing allows you to distribute the load on a particular service or services over a larger array of resources. It’s basically making your service, as a whole, more tolerant of failure by being able to efficiently make use of all available resources.

If you are running any kind of high availability service over the Internet you need load balancing. Though, even small applications with just a few thousand users can benefit deeply from load balancing, as well. The only potential down-side is that you may need more than just one node to it. This isn’t always necessary as load-balancing can come in many shapes and sizes. For example, you might be doing load balancing on the same host node using multiple guest nodes on the same machine. All of the major services you probably use on a regular basis like your email, search engines, or popular social networking apps all make use of load balancing because it keeps things running a lot more smoothly as the number of users grow. If you’re not on-board with this yet – you probably should get on board quick.

How Do I Use It

There are few broad categories you can place load balancing techniques in. The easiest form of load balancing relies on existing system already built on top of how most systems function over the Internet (or large networks in general) and that’s DNS. DNS is a distributed system so it relies on multiple components in the network to do their job in order to make things more efficient. It reduces bottle-necks like those created by routing enormous amounts of packets across the planet in fractions of a second. Like most complex systems everything starts off small and simple and grows both horizontally and vertically, but at the core the protocols are fundamentally very simple.

DNS Load Balancing is simply relying on the DNS system to take care of the most basic problems for you. The way this works is you set the DNS record for a particular domain name to multiple IP addresses (usually one for each server) using low TTL (or Time to Live). Since DNS is cached at various levels this makes things like geographical loads efficient for services like name servers. A name server tells the DNS where to send the request for a particular domain name and can route packets to different locations depending on the geographical origin of the request thus alleviating network latency and allowing packets to travel shorter distances. Once the request comes in and is routed effectively the DNS is cached at multiple levels so that future requests are made to the same place. This can be cached at the local level, the ISP level and other levels in the parent zone. The name server then doesn’t become a bottle-neck since not every single request has to rely on that name server entirely. There is a TTL involved that will let the caching servers know when the cache has become stale and that it’s time to refresh. Also when requests to a particular server are no longer getting through the DNS server will know to try a different IP. So if you have different servers with different IPs in the DNS record that ultimately means if one server becomes unresponsive (potentially having gone down) the load is directed to a different server. The inherent problems with this approach are that it isn’t making very efficient use of all of your resources. It doesn’t take into account which servers are currently busy and if the DNS record has already been cached to a server that is now down you end up potentially being stuck with a poorly responsive server until the cache is refreshed. Additionally, you are exposing your infrastructure to the outside world by revealing the public IPs of your servers with no way to control the flow of traffic to an internal network. It’s very easy to have an unstable system this way. Most services that use this approach are usually just creating what is known as mirrors (servers that back each other up so that in case one goes down a backup can still be reached).

Software Load Balancing is another approach to solve some of the short-comings of the DNS offloading techniques described earlier. Software load balancers attempt to keep track of the available resources and when an incoming request is received it determines how to best allocate those resources in-order-to service that request. The benefits of this technique are that you don’t have to reveal your network setup to the outside world. Everything can be done on the internal networking configuration setup (whether that’s a local area network or otherwise), or in other words, you won’t expose your communication channels directly. Also, you have a tighter hand on security and distribution since you can more easily control the flow of traffic over the network. Some examples of common open-source load balancing software are Pound, Varnish, mod_proxy for Apache’s httpd, and Gearman. There are all sorts of nifty ways to balance the load across your network. You can have the load balancers poll the servers and check on resources like CPU usage, available memory, storage space, network traffic or open TCP connection, etc… The load balancer can then use this information to figure out how to best direct the incoming requests and serve up the responses as quickly and as efficiently as possible. There are still a few problems inherent to this technique depending on how you use it. If you’re only relying on a single machine you have a single point of failure. If the host node goes down the load balancer and all of your resources go with it. If you’ve only got one load balancer and multiple servers you still have a single point of failure. Additionally the load balancer itself can be DoSed given an attack of enough magnitude and proficiency. Not only that, but you have to worry about things like session storage consistency across multiple servers, file-system access, database synchronization between different database servers, and some network bottle-necks that might not always be easy to resolve with load balancing – to name a few.

Hardware Load Balancing there are some hardware load balancers as well. You can actually buy very expensive firewall/routers that take care of many of these things for you. Most people usually just setup a dedicated node or two with software load balancers that pretty much do the same thing. These hardware load balancers might do a better job of handling security and high bandwidth loads like Cisco’s ASA, but they do come with a heavy price tag.

Some Load Balancing Tips

There are some pretty common approaches to some of the problem inherent to distributing a service over multiple servers. For example, take your session storage as the most obvious problem. If you’re using PHP you are probably using the built in session handler, which makes use of file-based sessions. If you have users being directed to different servers by the load balancer you end up with the user having multiple sessions across those servers (that might be a little problematic for your application and annoying to the user). Some people will try to avoid this by creating what’s called a sticky session. Once the session is generated for that user they’re sent a cookie that lets the load balancer know upon subsequent requests to direct the user to this particular server. There are a few minor problems with that, but nothing you couldn’t work out through a well-planned architectural approach. Another way to approach this is by creating a centralized session storage server where all the requests will look for the session. Depending on your infrastructure this may or may not be a good idea and keep in mind it also creates a single point of failure. For example, if your servers are built on stacks (you have several software-based servers running on the same node like a webserver, database server, application server, etc…) it takes some tinkering to configure each stack to work from a centralized session storage. You can use something like Redis where you can have master/slave replication across all stacks. This takes a little less configuration and puts the dynamic into the software stack layer – thereby removing it from the load-balancing layer.

The other obvious problem is file system storage. If you allow your users to upload files to your server, or you store large amounts of files that your application relies on heavily, there needs to be some system whereby your application layer can access those files considering the load balancing may send requests to different servers. Again there is a centralized approach like with session storage, but even with a replication approach – to avoid the single-point of failure down side – you might create the problem of over redundancy. If your servers are set up in stacks having four or five copies of each file (or more depending on how many servers you have) on each server stack is a bit of a waste, especially if you’re already using RAID arrays for redundancy. Even if you have a centralized set of servers for storage you still face the problem of network overload. For example, consider that if your backbone bandwidth capacity is at 100Mbps but your central network bandwidth capcity is at 10x100Mbps you eventually create a bottleneck with increased usage as your backbone can only serve up to 100 megabits per second of traffic at any given time.

Using a CDN (or Content Delivery Network) is one solution often used when large amounts of files need be shared across a network, but this can also be a bit costly depending on your needs. In its simplest form a CDN is really just a group of servers that store files or data objects for you and replicate them across multiple nodes allowing many other servers on the network to access that data with improvements in caching and high bandwidth to reduce latency. The servers in the CDN clusters are usually strategically located on the edges of the core network to minimize the bottlenecks involved in the centralized network loop. So you are redirecting the traffic to access file storage away from the central network and off to the edge servers expanding on bandwidth and minimizing on bottle neck traffic. This solves both the single-point of failure problem as well as taking the complexity mechanism away from the server stack which can ultimately help reduce loads and create more efficient load balancing. Most services that utilize CDNs are usually ones that need to offer high-bandwidth access to a large user base with consistency. For example, a service that offer Hi-Definition video streaming, large photo sharing web sites, or other media services with high availability needs. You don’t always have to build this infrastructure yourself. You can rely on services like amazon Cloud Front which is a pay-as-you-go CDN service offered by amazon. There are many other competitors, of course, that can offer cheap CDN solutions. Depending on the sensitivity of your data this may or may not be an option for your particular SaaS needs. Still something to consider.

Besides just file storage you probably have a lot of database concerns in a system that scales horizontally, as well. If you’re just using a single LAMP stack with little more than PHP, MySQL and Apache running your back-end it might seem easy to scale wide at first. The problem you’re likely to run into head-on is the data-replication across your MySQL servers. The database is almost always the biggest bottleneck in SaaS. It usually contains tons of data that virtually every one of your users will access with each hit. There’s only so much traffic a single database server can handle, but setting up two or more database servers can show some significant improvement. Your load balancer can also play a role in this. There can be data object caching mechanisms in place to ease off some of the load for the most frequented queries. There can also be network latency issues to deal with once you have several database servers all replicating (especially if these servers are geographically spaced out across different data centers, cities, countries or even on different continents). Chunking is definitely not something I’d advice. It throws way too many variables into the equation and presents more problems than solutions – for the most applications.

The truth is whoever you ask will tell you their language is the easiest, best, most powerful, or whatever reason they can think of for you to learn and use that language! As a joke I posted this video on youtube along with others about individual languages, but this one seemed to get the most hits. Is it that people are very inclined to find the one ultimate programming language or just laugh about the rest? Who really knows, but it’s funny…

This is kind of like asking a multilingual person which language they think I’ll find the easiest to learn. While there may be some valid answers, most of them will probably be subjective and they won’t answer the true question one should be asking. That is, what am I going to be using the language for. Just like when you decide to learn French, because you’re either moving to France, or would like to communicate with someone who speaks French, similarly, you pick a programming language because you would like to communicate with a computer in a way that meets your objectives. What you intend to do with the language, and what it can do for you, however, may not always be so apparent at first.

I only recently realized how truly complicated it may be for someone who does not come from a programming or technical background to actually chose their first programming language. Having recently examined a massive comprehensive list of programming languages on wikipedia I found that there are currently over 600 programming languages to chose from and that doesn’t even include the more than 300 dialects of BASIC and some other various esoteric languages. This also probably doesn’t account for some of the lesser-known dialects or derivatives of some of these languages. Since not all programming languages have an official specification they very well may be implemented in dozens of different ways in smaller niches.

If we look at programming languages broken up into categorical or even chronological lists the information still doesn’t help make the choice any easier; or even more useful. However, if you simply attribute languages to their strongest generational origins you can narrow the list down to just a few dozen languages. If you take away some of the older generations and put emphasis on those languages highest in popularity and active use/development you come up with just a few languages and their respective dialects. However, this still isn’t informative enough to help someone decide their first language so I split up languages based on their strongest usage and in the world of computer science this comes down to two broad categories (systems programming and utility or application programming). The most notable distinction between these two types of programming is that systems programming aims to provide software to communicate with hardware while application programming usually aims to provide software for the user that’s sitting at the computer. So while software like a text editor or word processor is considered application software, software like a disk formatting/partitioning utility is considered systems software. Your operating system has to deal with the hardware in your computer directly in order to provide application programs with a means to do things like write to your computers memory while the operating systems page table and memory manager can control how this hardware is being used by the various application programs.

C

By far, still one of the most popular programming languages still around and even though it is still used by many to develop application software it hasn’t lost its popularity or its power.

C is not a language you usually pick to write every-day utility applications. If you chose to start learning C be prepared to start learning a lot of other systems programming concepts and technical hardware documentation as well. Most Computer Science majors take C as one of their first programming language courses in college. This is important, because there is a huge amount of software that’s written in C. For example, most operating system software is either written in C. There may be some C++ in there, but for the most part you’ll find a lot of linux distributions are made up of a huge amount of C code and much smaller portions written in either C++ or some other similar language. You may hear about Assembly language as well when learning or working with C. Essentially, when a C program is compiled into a native binary and run as an executable program it is technically transported to Assembler. You take a high-level language like C and, eventually, to get it to run on the machine it has to become low-level software in bytecodes the machine architecture can execute. C is still a high-level programming language, but it has also been recognized for its lack of agility in relation to programming languages like assembly which is a low-level programming language. Don’t let this confuse you, however, C is a powerful language and in fact many of the popular languages you will likely hear about or discover in this article were written in C. For example, PHP, Python, and Java are languages whose APIs and extensions were written in C.

However, C can be tough. Writing non-buggy C code is costly. It can take a lot of time, because you either have to find the libraries you need and implement them or write them yourself. C is a procedural and somewhat imperative language. It also teaches concurrent programming and programming with side-effects, which is very different from languages like Scheme where you program without side-effects. C programs are like one big global scope where everything can effect everything else. So you have to be very careful about managing your memory in C. You have to worry about pointers and references and data types everywhere in your code. You have the basic constructs like IFs, and loops, and functions, but ultimately you have to learn to do a lot of things other programming languages can make a lot easier, because they already have extensions that implement a lot of these popular C libraries built right into the language.

So, unless you plan on designing an API for a larger program or build some system utility C may not be the right language for you to learn. If you’re a compsci major you’re probably going to learn it as your first language whether you want to or not, but lets face it you chose the degree…

BASIC

BASIC has been around for quite a while as well and it has hundreds of dialects. It was popularized by many hobbyists during the 80s and grew further in popularity on Windows during the 90s with Microsoft’s Visual Basic suite that attempted to keep the language as simple but as powerful as possible. BASIC is not very difficult to learn, but it is also a compiled language like C and has declined in popularity over the last decade. It might not be the best language to work with, but it is still high on the hobbyists list. Much like languages that were once popular to learn just as a hobby and were fun to play with (like LOGO which was a dialect of LISP) not many people take it seriously.

BASIC has the essential control structures you’d find in almost any language like IFs, loops, and GOTOs, but it was fundamentally built on the concept of sequential programming where the entire program is built on one huge sequence of instructions. There are subroutines (like functions) and some dialects implement a lot of other modern features, but for the most part it’s great for when you want to learn programming for fun. If you’re serious about building cross-platform or enterprise-level applications BASIC is far from a first choice.

Java

Java and its other Java-based languages stand out for their compile-once run anywhere trait as opposed to many other compiled languages where you write the code once and then have to compile it for each different platform you chose to run it on. With Java, if you chose to compile your code to Java bytecodes to run in the JVM you will only need to compile it once. The JVM (or the Java Virtual Machine) can pretty much run on any platform (Windows, Linux, MAC OS, etc…) and works with the systems hardware directly through its VM. This enables programmers to be able to compile their Java code on any machine just once and it will run on any other machine in virtually any platform without having to recompile for that specific platform. Java can also be run in JRE (or the Java Runtime Enviornment) so it works as an interpreted language as well. Java’s popularity hasn’t declined much over the years and it’s gained quite the reputation with later adopting open source initiatives.

Java is also popularly taught in compsci courses in colleges, institutes, and universities around the world. It’s similar to C in that it is a statically typed language and has functions and basic loops and other constructs. However, Java is an object-oriented language. C is pretty much procedural in paradigm. You can build structs and things in C, but Java makes abstraction a whole lot easier with its OOP features. You can get a whole lot more done in development in a fraction of the time it might take you to do the same in C. So developing day-to-day applications in Java is a lot more common than with C. It’s just that a lot of the folks that have learned C and know it well have stuck to it over the decades and continue using it. Java is a much newer language. It appeared around the mid 90′s, but it has proven itself in the last 16 years or so. C has been around since the early 70′s and hasn’t changed much. The most current standard of C is C11, its predecessor was C99. Java is at Standard Edition 7.

Java is also considered a fast and secure language for a number of reasons. It is skeptical whether or not all of these reasons hold true, but for the most part they’re built on some solid grounds. First, Java code runs in the JVM, or the Java Vritual Machine, which means the VM can check the compiled bytecodes of the program and make sure they’re valid Java bytecodes before running or executing the code. Second, Java code is cross-platform so it easily translates to the same machine code across different platforms without much concern over the implemented libraries. Java is expected to be very performant because of its JVM. This means your Java programs run directly in a virtual machine that sits on top of the hardware layer allowing direct hardware implementations and interfaces as opposed to some other VM concepts where the program runs in the VM that runs on top of operating system or its implemented libraries that runs on top of the hardware. There’s somewhat of a more direct interaction there. Between Java and some other scripting languages like Perl, Python or PHP – this might be an advantage, but between C and Java it can go either way. In most cases C would easily out-perform Java, but in a few cases it might go the other way around.

PHP

PHP is probably the most popular language on the web. It has many followers and a huge open source community. It’s an interpreted language that was originally developed for producing dynamic web pages. However, today it is seen as a general purpose language. What makes PHP so great is that it works very well with web servers. You can install it as a web server module or run it on the command line. It has many useful built-in features that make web development easier right out of the box. PHP is also built on share-nothing architecture so it scales very easily and doesn’t require much configuration. It offers automatic memory management and it’s somewhat loosely typed so its data types may not be very suitable for edge cases, but that can be debated. For most general purposes PHP works great, but like BASIC it attracts a lot of hobbyists given that it lowers the bar of entry.

Unlike with C, in PHP you do not have to worry about managing your own memory. You can easily build data structures, facilitate external resources to databases or other libraries directly through the PHP extensions, and generate output to standard streams without a lot of fuss. It’s easy to take a general idea and implement it in PHP very quickly. Most people do this with Python and Perl as well to get a working prototype up and running. However, if you build a lot of prototypes, you know that they end up getting tossed out when you start building the real thing. Regardless, PHP is a great language to get code working quickly and very similar in syntax to languages like C and Perl. However, the down side is that these languages are also considered very ugly and have many extensions with poor implementations or interfaces or leaky memory. Not everything about PHP or Perl is great, but it works. At the end of the day it takes a fraction of the time to write PHP or Perl code that would do the same thing in languages like C and with less possibility of bugs since these languages are usually very forgiving and try to account for user error where possible.

PHP is extended by C and is built around the Zend Engine, which is the PHP Virtual Machine. PHP has different SAPIs, or Server APIs, for different web servers and platforms. Among the most popular are probably the Apache httpd module, which is known as mod_php and the fastcgi /fcgi SAPIs. The difference between the two is basically like running PHP inside your webserver as a part of the webserver program (mod_php), and running another program along-side of your webserver that interfaces with it through a CGI (Common Gateway Interface), which is what the cgi/fastcgi SAPIs are built around. There are lots of different implementations, but the module running as a part of the webserver usually trumps the others in performance and scale. PHP also has a CLI SAPI, which allows you to run PHP directly from the command line. You could use this to build command-line scripts like the popular BASH scripting language, on *nix shells. However, most people don’t use PHP to build command-line programs. It’s not the most performant programming language, but it works well for things like the web where you want to build dynamic websites or applications. Just tiny programs that execute for a very short period of time and run independently of one another. When you look into building things like long-running daemons, you usually turn-away from PHP and head for languages like C or even Java.

Other General Purpose Languages

There are many languages considered for both web development and as general purpose languages that are also dynamically or loosely typed and offer automatic memory management and even web server modules just like PHP. Languages like Python, Perl, and Ruby are also exceedingly popular and quite similar to PHP in many ways though they are not all based on the same generational languages. Of course shell scripting is also going to fit under general purpose in most cases and so Bash, sed, AWK, etc.. are also great languages to know.

To some people’s surprise, javascript is now becoming somewhat of a general purpose language itself. Recent VM implementations like Node.js make using javascript faster and a little more powerful than some of its earlier ancestors. One of the best things about javascript is it’s non-blocking nature and event-driven capabilities. It’s a great language for automating event-driven tasks by setting up listeners and such. It’s got a lot of uses on the web and offers multiple paradigms as well.

Beyond

Beyond just looking at what all of these programming languages can do for you it’s important to realize one language isn’t always enough to do what you need. If you’re going to start learning a programming language it’s easier to pick one that won’t require a lot of time to setup and configure. Something llike Python or PHP or even javascript is easy to just install and start writing code and the best part is you can just run that code instantly without having to compile anything and see the result right away. These languages aren’t very hard to learn because they have a lot of free online resources, documentation, and a lot of people already use them so you shouldn’t have too much trouble finding quick tutorials or examples of code that show you how to write short and useful programs. But of course your mileage may vary!

Over time, when you have learned your first programming language very well you may find the need to do some things that aren’t always very easy or even possible with that language (or you may never experience this depending on the language and what you’re doing). This may lead you to start using another language in place of or along side of that language for a similar project or a different project. If you’re a hobbyist doing this for fun you might not be so inclined to learn more languages, but if you’re a professional you will probably need to learn many languages over the years. It doesn’t hurt to have a long list of programming languages on your resume for a job and it certainly won’t hurt to already have some experience with a language you’ll be using on a new project at work. However, most programmers will be quite proficient in just two or three languages and have some overall understanding of others. This is usually all you need in the majority of cases.

Some Things Never Change

Looks Can Be Deceiving

Sometimes looks can be quite deceiving, though. For instance, when you consider some of the raw numbers that these companies claim – head held high – like the number of users they serve, or the number of people they employee, or the amount of revenues they generate, not all of it is put into clear perspective.

For eaxmple, facebook, recently claimed it had 800 million users. This is indeed a huge number of users. If you were to compare it to the average website on the web today or even the average corporation, this makes it seem like a huge company. But facebook only employs around 2,000 people and none of their users pay a single dime for using the service. The service is free so clearly 800 million people aren’t facebook’s customers. Then again what does facebook actually do for its 800 million users? Well, we know a lot about what they don’t do. They certainly don’t write all of the software that their users come back for so often. There are tens of thousands of facebook developers that either develop facebook apps or build on top of the facebook platform just for the benefit of having access to the huge social graph that gathers under the umbrella that is facebook. So nothing about their software actually makes their service unique per se, just that they have access to a huge userbase which interests a lot of parties. They also don’t do much of anything significant with what software they have built. If we consider their largest and probably most significant work it would have to be Casandra. HipHop might probably rank second. Their SDK is actually pretty much poorly documented and poorly supported, for the most part. Photos? Well, they do claim to have the largest photo sharing application on the web, but then again flickr isn’t far behind. And it’s not like facebook photos does anything really interesting with the actual photos apart from tagging (clearly their most popular feature). Other sites have actually done a far better job with photo sharing than facebook even though they may not be as popular. So in all reality these numbers – once put into some perspective – actually indicate pretty negative things about facebook even though it tries to put them in a positive light with a flashy spin (in hope that this will only boost their already growing popularity of course). The reality, however, is that this only indicates how under-staffed, under-paid, and over-commited facebook really is.

Here are some charts to demonstrate what these numbers actually indicate with some contrast for visualization.

Even if we consider every single user on facebook to be a customer of facebook, as a company, this would mean they are gaining the attention of about half the number of customers Microsoft currently attracts with it’s dominant Windows Operating system market share. They are neck and neck with Google in terms of users, but facebook claims to have surpassed Google in pageviews. There are probably very good reasons as to why that is. Let’s not forget people who visit facebook are usually their to do a lot of things. Chat with friends, browse every photo their friend ever took of their toe-nail, armpit, mirror-shot, bar-hoping-night, passed-out-magic-marker-art, and just about every other pointless moment their friends have ever captured on camera and uploaded to facebook for the world to see, not to mention the hundreds of wall-posts, messages, and I-Like hits the site gets every day from all of its users. In a single second facebook will probably have received around 2,000 photo uploads from its users. In about the time it has taken me to write this blog post they will probably have gotten around 3 Million new photo uploads. Each one of those uploads will generate a page view. Each time you play farmville or use an app that lets you know when your friend made a funny face or tried to some-how through the powers of dark magic poke you over TCP/IP that also generates what facebook considers a page-view.

The point I’m trying to make here is that facebook is clearly a content-provider. But just like Google, they aren’t the ones producing all of the content. However, unlike Google, facebook does actually facilitate a place for you to store and share all of this content and encourages it profusely by not setting a lot of stringent restrictions on how much content you can share.

If we took the amount of content users shared as an indication that a company was powerful in reach, however, we would have to say Hotmail is probably more of a social networking tool than facebook and twitter combined. Consider that Hotmail has only around 364 Million users and they share billions of emails every day – this would indicate more active sharing than facebook has ever openly claimed – even though Hotmail has less than half the userbase. Additionally if we compare the number of developers that actually work on supporting the service that these users are making use of facebook clearly has a higher developer to user load ratio than any of these other companies.

Stepping Outside of the Browser for a Minute

So it’s not clear that twitter, facebook, myspace or any of these other very popular social networking sites, are actually proving to be nearly as profitable as they are popular, despite them clearly being a great tool for people to communicate. The reasons for this are mainly the advertising risk. Twitter, facebook, myspace all clearly strong in numbers (any company that has to support a site that has users in the millions is nothing to laugh at), are still weak in strategy. There’s no good reason for a serious advertising (with billions of dollars to invest) to make long-term commitments to a user base that may very well disappear in a few years.

If you think of what Twitter is really all about, for example, it’s like reminds me of when we used to pass short little notes around in class during high school on tiny snippets of paper with messages like “see you at lunch…” or something silly like that. You basically send messages to people who choose to read them in tiny bite-sized pieces. This is nothing fascinating. The only reason the service is even popular is because a lot of people – at one point – found their friends using it (just  like myspace and facebook evolved) and decided to use it as well. That’s the thing about these social-based sites. They are easy to gather around and just as easy to turn away from. Eventually some one gets bored and a collapse of the social graph brings down the whole thing.

Google, on the other hand, doesn’t actually want to keep you within its borders all day long. It actually wants to get you the information you were looking for and out the door as quickly as possible with little to no distraction along the way. Quite the opposite from what facebook aims to do, because facebook understands it’s only value is in how long its users chose to stay. So they will do everything in their power to keep their users there longer. Google knows its users only come back because they know something else is out there to be found and that Google is probably going to find it for them more quickly (since it’s done so numerous times in the past with great success).

If you’re looking for a nice read on the subject of Google – by the way – I’d recommend this title right off my recent bookshelf Googled The End of the World as We Know It by Ken Auletta

Googled

It’s Not Browsing, It’s Searching

It makes sense that someone would turn to their friends or colleagues, etc, to find out about which of the latest blockbuster movies is popular when deciding on purhcasing a DVD or paying a visit to the movie theater, for example. But it might not make a whole lot of sense that someone would spend their time searching facebook when they’re looking for information on World War II history, or a photo of the point contact transistor at Bell Labs in 1947. People are probably more likely to turn to Google or Wikipedia for that type of search. Even though I have no doubt people are probably starting conversations on facebook groups or posting messages on people’s walls with very similar questions as you read this. To be fair, however, facebook and twitter probably do have good uses in searching for information as well. For example, you might hear about some new trend or some recent local, national, or even global event that just took place on one of these social networking sites, but it’s not like you can’t hear about that on the news either.

It’s just people’s naturally insatiable curiosity that won’t stop them from asking questions or looking for answers. This is why Google has become so popular today. They found the ultimate way to keep users coming back for more. Facebook, while it appears to have done the same thing, it apparently does it for all the wrong reasons. Google’s mission is to solve the problem of search (people are always asking questions about something) by making use of the worlds information as it becomes available (i.e. the web, books, news papers, television, etc…). However, facebook’s mission (while not even clear at this stage) seems to publicly say it’s attempting to do something fancy with something called the “social graph” (yes I really put a quote-unqoute on that). However, internally, facebook developers aren’t really sure what they’re doing. They seem to blog a lot about all these neat new technologies the company is getting its hands on, but what about this social graph? What does it do? How does it really make our lives better? Because we can communicate with our friends? No, that can’t be it. We’ve been communicating with our friends long before facebook ever showed up. Is it because facebook makes it easier to see what our friends are doing? No, that can’t be it either, because our friends still have to show us what they’re doing for us to see it (so instead of you seeing your friend getting drunk at the bar you get to see them in a photo on facebook drunk at the bar). Clearly, that isn’t making anyone’s life better (at least not yet). Is it because facebook offers a platform where you can connect millions of people and extract large quantities of useful information out of this so-called social graph? Well, to whose benefit is that, exactly? I don’t know about you, but I really don’t want everyone having access to all of my information just because I want to use their app to see what books my friends are reading (and yes there are very little to virtually no restrictions on how much information each app you use can access about you). I really don’t care to know how many people poke how many other people every day instead of getting off the computer or pulling their eyes away from their smart phone for a minute to take a look at the people in their immediate presence for a change. I don’t find anything about that useful, do you? Perhaps some might, but who? It’s probably not the average joe. It would probably be someone with a serious vested interest in you. Someone like say a company that produces hundred dollar designer jeans and seeks yet another way to pry into your personal details to figure out just how much influence they can muster to get you and all of your friends to buy those really expensive jeans that will just make your butt look spectacular.

At the end of the day, the numbers speak for themselves.

But, you’re just browsing the web. It’s not like any of this occurs to the average person on a regular basis as they “just browse the web”…