HandBrake Wrapper Script

I've been encoding my DVD's lately using HandBrake for Linux. As far as I can tell, there isn't a GUI for the Linux version, though there is a very nice one for OS X. However, you don't need the GUI at all.

Once you have compiled HandBrake from source, you have everything you need. Except that the default options in the CLI program (HBTest) aren't exactly the greatest in the world. So every time you want to encode a movie, you have to type a long string of command line options which aren't strictly necessary, since they're the same every time. (I'm assuming you encode all your movies the same way, because that's what I do ... and who wants to worry about encoding each movie differently?)

I encode the video using the x264 codec at 700 kbps, and the audio using the faac codec at 96 kbps. I find that this gives a high enough quality rip that the artifacts aren't typically very apparent. In most cases, my encoded version is indistinguishable from the MPEG-2 version on the DVD, except for the small size.

Normally, I would have to type:
/home/sean/Desktop/HandBrake-0.7.0/HBTest -e x264 -E faac -2 -b 700 -B 96 -w 512 -i /dev/dvd -o This\ is\ the\ movie\ title.mp4

That's a lot of typing, especially considering the fact that all of it is the same every time except for the -o argument.

To save myself some time (it's a valuable couple of seconds!), I wrote a wrapper script in Python to use my defaults instead of HandBrake's.

Here it is.

Now all I have to type is:
./hbencode.py 'This is the movie title.mp4'

Which of course is a lot faster.

I also decided that I might at some point forget to type a title for the movie, but I'd still want the movie to be ripped and encoded, and I could name it later. So my solution was that if nothing is passed in at the command line, it would get the current timestamp and use that as the filename. That should be unique enough that I'd be able to repeatedly encode things without setting a title and be able to go back and set their names later. It has the added bonus of keeping the files in chronological order, according to when they're ripped.

The next step is to take more command line options than just the movie title, so that the defaults can be changed if necessary. The first addition would be the ability to set the title to rip, in case you're ripping a TV show, or a movie where the main feature is (for whatever dastardly reason) not the first title.

This is a very simple script, but it makes my video encoding life a little simpler, and a little more pleasant. Hopefully it does the same for you.

Web Based Remote Control

I spend a lot of time at my computer, but I spend even more time near my computer but not quite at the keyboard. It's a Mac Mini, and most of the time it's playing music in iTunes or a movie in VLC. I wanted a way to control those applications remotely, so I could pause without going all the way over to the computer. Since I have a Nokia 770, I thought the easiest way to do this would be to write a web page that can control these applications, and connect to it from the 770.

The first thing I needed was a way to control the applications. This was quite simple, of course, using Applescript. I made a small script for each action I wanted to perform. For example:

tell application "iTunes"
   playpause
end tell

All the scripts I wrote were similar in complexity, but more complicated scripts can of course be written.

The next step was to run these scripts from a web page. PHP handles this quite nicely, using the system() command. However, it won't work right out of the box, because OS X runs the web browser as a different user than the person running iTunes (for obvious security reasons). So I had to add a line to the /etc/sudoers file:

www ALL = NOPASSWD; /usr/bin/osascript

This gives the user running Apache the right to run osascript (which executes Applescript files) without typing in a password. This is probably not the best thing to do, security-wise, so it's pretty important to make sure that your machine is only accessible to computers on your local network.

Next, the PHP. It's quite simple. I only need one page, which takes a script and executes it as if it were on the command line. Here it is (execute.php):

<?php

$script = $_REQUEST['script'];

system($script);

?>

I'm planning on sending requests to this PHP file via XMLHttpRequest, of course. I'm using my standard xmlhttp() function which you can get from a previous blog. The only other Javascript function I actually need is the execute() function, which sends a request to execute.php. Here it is:

function execute(script) {
   var req = xmlhttp();
   var param = 'script='+script;
 
   //this should be made async
   req.open('POST', 'execute.php', true);
   req.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded');
   req.send(param);
}

This is a pretty simple function, sending a single parameter asyncronously. Since it's only sending the request, and doesn't really need to return anything, I don't even bother checking the return. We just shoot off the request and wait for more input. You'll know if it worked it the music starts playing.

It was important to me that I didn't limit this program to any set of functionality that would be difficult to augment later. So I created an XML file that holds records for each program that can be controlled and the scripts to run in order to control it. Originally, I wanted the interface to be built dynamically via Javascript, so I crafted the XML file such that it had no attributes, only elements, because my JS XML parser doesn't read attributes (I should work on that). Unfortunately, it seems that the browser on the 770 doesn't support dynamically created DOM elements, so this approach didn't work. Plenty of wasted work there (and a lot more Javascript than I've included here).

This was when it dawned on me that I don't actually need to build the interface every time the page loads. It only needs to be done when the XML file is changed. So I wrote a Python program to take the XML file, parse it, and output the same HTML as would have been produced by the Javascript. I recreated the XML file to use attributes instead of just elements, because I find that to be simpler to read, and it's much easier to handle that in Python's xml.sax library. Here's the Python (buildinterface.py):

import os, sys
from xml.sax import saxutils, handler, make_parser

#set up the environment variables
config_file = "server2.xml"
html_file = "index.html"
output_file = file(html_file, 'w')

#globals programs dictionary
programs = []

class Program:
   def __init__(self, name):
       self.name = name
       self.commands = []
     
       programs.append(self)
     
   def addCommand(self, cmd):
       self.commands.append(cmd)
     
class Command:
   def __init__(self, name, script):
       self.name = name
       self.script = script
     
class ConfParser(handler.ContentHandler):
   def __init__(self):
       handler.ContentHandler.__init__(self)
     
   def startElement(self, name, attrs):
       global temp_prog
       if name == "program":
           temp_prog = Program(attrs['name'])
       elif name == "command":
           temp_prog.addCommand(Command(attrs['name'], attrs['script']))
         
def write_html(out = sys.stdout):
   out.write("<html><head><title>Remote Control</title>")
   out.write("<link rel='stylesheet' type='text/css' href='remote.css' />")
   out.write("<script type='text/javascript' src='remote.js'></script>")
   out.write("</head><body>")
   for prog in programs:
       out.write("<fieldset><legend>" + prog.name + "</legend>")
       out.write("<div id='" + prog.name + "_div'>")
     
       for cmd in prog.commands:
           out.write("<span class='link' onclick='execute(\"" + cmd.script + "\");'>" + cmd.name + "</span>")
           out.write("<br />")
         
       out.write("</div></fieldset>")
   out.write("</body></html>")

parser = make_parser()
parser.setContentHandler(ConfParser())
parser.parse(config_file)

write_html(output_file)

output_file.close()

As you can see, it outputs directly to index.html, which is convenient. Every time you run buildinterface.py, you have a fully updated interface waiting for you to load up in your browser. And in case you were wondering what those span tags are all about, they are to mimic links without the bother of actually putting a link on there. Here's the CSS for that:

.link {
   text-decoration: underline;
   font-size: 10pt;
   color: black;
   cursor: pointer;
}

Each link has its onclick event set to call execute() with the script to run. It's a fairly simple scheme, and when I loaded it in my browser to test it out, it worked swimmingly. I could control iTunes, VLC, Quicktime, and the system volume. I pulled out my laptop and connected, and found that it worked from remote systems. Now it was time to try it out on the 770. I fired up the browser and opened my page. This time the interface loaded, which was of course promising. However, none of the links worked. So the remote control program works, and is extensible, but I haven't been able to get it working on my Nokia. It seems that the browser doesn't want to send the XMLHttpRequests, which is weird, because it's Opera, and I tested this in Opera on both OS X and Linux, and it works there. I'll be trying to get it to work on the 770, but until then, here's the working code.

Things that could be added:
1. Get it to work on the Nokia 770.
2. Write plugins for other programs, to build out the capabilities of the remote.
3. Have the PHP return something, so it's possible to tell if the command has been executed or not. This will allow for more interesting possibilities on the front end.
4. Anything else ... ?

You can download the code here. Let me know how it goes, or if there are any problems you find with it. And let's here about those plugins!

Associative Array Doubly Linked List

This isn't going to be a demonstration of an application of any kind, but rather I'm going to show a data structure I developed the other day to help out with my web applications. I implemented it in Javascript, but it should be trivial to port it to other languages. (If the language doesn't support associative arrays on its own, then you'll have to implement that first.)

The idea behind an Associative Array Doubly Linked List (AADLList) starts with an associative array and adds a doubly linked list to it. In an associative array, you have a bunch of key/value pairs, where the value is accessed by its key (normal arrays can be viewed as an associative array where the keys are integers starting with 0). In a linked list, you have to define an object that has not only its data, but also a pointer to the next object in the list. A linked list will also have a pointer to the front of the list. A doubly linked list takes this a step further, by giving each object a pointer to the object before it AND after it, and the list must have pointers to the first and last items in the list.

I wanted to make this data structure independent of the application you're going to use it in, so instead of defining information fields in the objects as well as the pointers, the nodes in this list will have another variable inside that you can fill with whatever object you want.

First, we have to define the list objects, AADLListNode and AADLList:

function AADLListNode() {
    this.node = null;
    this.next = null;
    this.prev = null;
}

function AADLList() {
    this.list = new Array();
    this.first = null;
    this.last = null;

Note that I omitted a trailing bracket at the end of the AADLList definition. That is because the rest of the code goes inside the AADLList class definition.

As you can see, the ListNode class has a node object and its two pointers. The List object has a list (which is an array) and its two pointers. At first, the first and last pointers are null, which means the array is empty.

The first thing we need to do is add nodes to the list. When we're adding the first item in the list, we have to take more things into consideration. When the first node is inserted, the list's first and last pointers have to change from null to the new first node. So we write our first function:

/*
    add a node that is the first and therefore only node in the list
    */
    this.addFirst = function(key, val) {
        var node = new AADLListNode();
        node.node = val;
        node.next = null;
        node.prev = null;
        
        this.list[key] = node;
        
        this.last = node;
        this.first = node;
    }

After our first node has been added, we can start adding more nodes. Since it's a doubly linked list, we have the desirable ability to add nodes to either the front or the back of the list. (Actually, we will be able to insert a node anywhere in the list.) So we write two functions, to add a node to the front or back of the list.

/*
    add the given key/val pair as the first element in the list
    */
    this.addToFront = function(key, val) {
        var temp = this.first;
        
        var node = new AADLListNode();
        node.node = val;
        node.next = this.first;
        node.prev = null;
        
        if (this.isEmpty()) {
            this.addFirst(key, val);
        }
        
        this.list[key] = node;
        
        if (this.first) {
            this.first.prev = node;
        }
        this.first = node;
    }
    
    /*
    add the given key/val pair as the last element in the list
    */
    this.addToBack = function(key, val) {
        var node = new AADLListNode();
        node.node = val;
        node.next = null;
        node.prev = this.last;
        
        if (this.isEmpty()) {
            this.addFirst(key, val);
        }
        
        this.list[key] = node;
        
        if (this.last) {
            this.last.next = node;
        }
        this.last = node;
    }

The code in these functions is pretty simple. We just need to set the prev or next pointers of the newly created node, and update the first or last pointers of the list. As you can see, these functions use the isEmpty() function to determine whether they should call addFirst() or not. We have to create isEmpty() for ourselves, like so:

/*
    returns true if the list is empty
    */
    this.isEmpty = function() {
        if (!this.first && !this.last) {
            return true;
        } else {
            return false;
        }
    }

Simple enough. If the first and last pointers are both null, then the list is empty. It should, in theory, be possible to also just access the this.list.length value and check that it is zero, but unfortunately, when you remove an entry in a Javascript array, it does not disappear. The key continues to exist, but points to null. So the this.list.length field will be incorrect if a node is ever removed from the list.

I often don't care whether my nodes are inserted at the front or the back, so I threw a convenience function in:

/*
    simple convenience function
    just calls addToBack
    */
    this.add = function(key, val) {
        this.addToBack(key, val);
    }

Now we can add as many nodes as we want, either at the front or the back of the list. But how do we access the nodes once they're there? One way is to simply go through the object's members and use them, but this is generally not an awesome way to go (if you really wanted to do it, you'd have to do something like this: myList.list[myKey].node.myField ... etc. Clearly not what you want to be doing every time you want to retrieve something from the list). So we can write a function that will retrieve a given node for us based on its key.

/*
    try to get the node for a given key
    return the node if it exists, null if not
    */
    this.getNode = function(key) {
        for (var id in this.list) {
            if (id == key) {
                return this.list[id];
            }
        }
        
        return null;
    }

But that isn't always what we want, because it will return an AADLListNode object, which you probably won't want to use in your application. So we write another, similar function, to just return the object inside the node (without the pointers).

/*
    try to get the node value for a given key
    return the value of the node or null, if there is no node
    */
    this.getNodeValue = function(key) {
        var node = this.getNode(key);
        if (node) {
            return node.node;
        } else {
            return null;
        }
    }

Note that if we were simply doing a linked list (or even a doubly linked list), this operation would have an O(n) efficiency (worst case), because we would have to traverse the list and check each node to see if it was the one we're looking for. But since it's also an associative array, we can access it in O(1) time, greatly speeding up our data access. This is good.

I'm sure you've noticed that what this is missing is a way to remove a node from the list. So we're going to add that function now. We need to make sure we check that the node actually exists, then change the next object's prev pointer to point to the removed object's prev, and vice versa. (Don't know how to explain that very well, except to use the term "switcharoo," which I think should cover it pretty well.)

/*
    remove a node from the list
    return true on success, false on failure (if the given key isn't in the list)
    */
    this.removeNode = function(key) {
        var node = this.getNode(key);
        
        //if the node doesn't exist, you can't remove it
        if (!node) {
            return false;
        }
        
        var tempNext = node.next;
        var tempPrev = node.prev;
        
        //if the node doesn't have next/prev, you can't change them
        if (tempNext)
            tempNext.prev = tempPrev;
        if (tempPrev)
            tempPrev.next = tempNext;
        
        //if it's the first node, set the next node to be the first
        if (key == this.first.node.id) {
            if (tempNext) {
                this.first = tempNext;
            } else if (tempPrev) {
                this.first = tempPrev;
            } else {
                this.first = null;
            }
        }
        
        //if it's the last node, set the prev node to be the last
        if (node == this.last) {
            if (tempPrev) {
                this.last = tempPrev;
            } else if (tempNext) {
                this.last = tempNext;
            } else {
                this.last = null;
            }
        }
        
        //null out the entry in the array
        this.list[key] = null;
        
        return true;
    }

As you can see, this is by far the most complicated function in the class (and it's not even very complicated!).

And don't forget that trailing bracket!

}

Now we have a working AADLList, which gives us all the advantages of a doubly linked list and an associative array in one easy to use package. You have O(1) access to any node in the list, but you can also traverse from the back or the front and add nodes in whatever order you want (although that isn't implemented in this version).

So what would you want to add to the AADLList to make it a little better?

• How about an insert() function of some kind, so you can decide where in the list you want a node to be added?
• Build on your insert() function and make the AADLList a sorted list.
• Give it some queue-like or stack-like functionality, with push() and pop() operations. (Who wouldn't want the ultimate data structure?)

Feel free to use it, add to it, whatever. Let me know if you have any issues with it, or problems using it or implementing new features for it.

Skype's Purchaser

I just read Cringely's theory about Skype's near purchase by Rupert Murdoch. He thinks it was to put Skype into play as a takeover target by another telecom company. His guess is that it will be either a mobile phone company like Vodaphone or NTT DoCoMo, or a cable company who wants to get into the VOIP market, like Comcast.

I must say I disagree with him. Comcast already has a digital phone service, which uses VOIP. What use would they have for Skype? I don't buy it that they want those 20 million subscribers to sell movies to. Why would Skype's customers switch to Comcast? Many of them don't have broadband, and more are international (and Comcast isn't). So they wouldn't be eligible for such downloaded movies.

And a mobile phone company? They'd be just as interested in killing Skype as the normal telecom companies. Skype can be used by any device that can connect to the internet, basically. People in coffee shops can use Skype on their laptops. People with PDAs can talk whenever they have wireless coverage, as if their PDA was one of those PDA-phones that cost much more.

That gets me to my counter-theory: Skype will be purchased by a company that wants to expand into a new market and sell their own product to go along with Skype. They will blanket the US and possibly Europe with a wireless signal (WiMAX?) and sell PDA-like devices whose primary use is as a phone, but it also inherently has full access to the internet.

Who would do something like this? I would guess that it would be either Intel or Microsoft. Intel wants to build and sell WiMAX transmitters and cards. They want to expand into more markets, including the home. Basically, they want to put an Intel chip in as many places as possible. They'd get their WiMAX chips in every consumer's laptop, PDA, and especially the Intel-branded PDA-device that goes with the service. They could offer such a service at similar cost to a cellular phone, available both in the home and on the road. They could take over the mobile phone market and the traditional telecom market it one fell swoop. And they'd make a lot more profit, because they wouldn't have to launch satellites like the cellular companies or invest in underground or expensive infrastructure. They would only have to put a WiMAX station or two in a few score major cities and get a good amount of bandwidth.

But Microsoft could do the same thing. They're looking for ways to grow the company, and they always love expanding into new markets. Additionally, they just announced that they're planning to acquire more companies in the near future to use their technology. Skype could be one of these purchases. Microsoft sells the mobile OS that Skype runs on. They'd love to be able to build their own phone and sell it to people directly, getting hardware revenues instead of software licenses. Of course, they would also continue to license the OS, so they're still getting those revenues. They'd use the same strategy that Intel would, but would probably push it a lot harder and market it a lot better. And they'd tie it directly into Windows, and the MS-phone would cooperate with it perfectly ... when you're sitting at your Windows Vista computer with your MS-phone in your pocket and you get a call, you see it on your screen (maybe it pauses your music or movie), and you can either answer it with your computer or use your MS-phone as the handset.

I can see Microsoft spending a few billion dollars to make that happen. If they did this (with their OS monopoly), they could rule the world for quite a while longer.

So I say be wary of MS looking into Skype. I'd much rather have Intel do it, or have Skype continue under its own sails. There's nothing wrong with a private company that makes $70 million dollars a year and has awesome growth potential. So there's no need to sell it.

Dashboard Escaper

So I've done a little work on the HTML Escaper I released here a few days ago. While it might be convenient to have a web page open that lets you escape your code for the web, it would be even MORE convenient if you could just do it in a widget. So I've decided to do just that.

The Dashboard Escaper is basically the same program as the HTML Escaper, except that instead of being rendered by your web browser, it is run by Apple's Dashboard.

I decided to make a couple of other changes: there is no longer a button to force it to escape the code. It does it automatically. It escapes the code once per second, and also any time a key is pressed while the source text field is active. I think this automatic, real-time behavior is more suited to the Dashboard concept.

Making the widget was actually quite easy. I already had the web application written, and just wanted to run/display it in a new way. I used a special Dashboard IDE called Widgetarium (link), which contains an editor for your Javascript, HTML and CSS, a widget packager, a Dashboard debugging emulator, and a source level Javascript debugger. Apparently you can insert breakpoints into your JS code, but I haven't tried it yet (I haven't put any Javascript with bugs in it into the editor or debugger).

After downloading Widgetarium and running it, I started a new project. I called it Escaper and went right along.

It asked me for the size of my widget, and I gave it 300x200, as well as the color. It automatically created a PNG to use as the background of the widget.

The widget was now ready to be written. The first thing I did was open Escaper.html. I had to change the line which includes the .js file, because it was including $_NAME_.js instead of Escaper.js. Then I just had to put in my HTML where it directed me to do so ... my Escaper.html looks like this:

<html>
<head>
<!-- The style sheet should be kept in a separate file; it contains the design for the widget -->
<style type="text/css">
 @import "Escaper.css";
</style>
    <script type='text/javascript' src="/System/Library/WidgetResources/button/genericButton.js"></script>
    <script type='text/javascript' src="Escaper.js"></script>

<title>Escaper</title>
</head>
<body onload='setup();'>
    <div id="front" onmouseover='mousemove(event);' onmouseout='mouseexit(event);'>
     <img span="backgroundImage" src="Default.png">
     <textarea id="source" rows="10"></textarea>
     <textarea id="resultcode" rows="10"></textarea>
     <div id="result"></div><!-- need to make this scrollable! -->
    </div>


</body>
</html>

I executed the widget by selecting Run from the Project menu, and my widget popped up in the pseudo-Dashboard in the Widgetarium environment. Unfortunately, it was blank.

I needed to fix this, and I discovered the problem by using the IDE's DOM inspector (on the right hand side of the application). I found that the elements I had defined were off the image, and so would not be displayed. I opened the Escaper.css file and positioned my elements so they would be sure to show up on the widget. Here is my CSS file:

body {
    margin: 0;
}

.backgroundImage {
    position: absolute;
    top: 0px;
    left: 0px;
}

#widgetButton {
    font: 10px "Lucida Grande";
    font-weight: bold;
}

#source {
    position:absolute;
    left:30px;
    top:10px;
}

#resultcode {
    position:absolute;
    right:30px;
    top:10px;
}

#result {
    position:absolute;
    left:30px;
    top:150px;
}

Note that I only wrote the last two rules. The rest are generated automatically by Widgetarium.

Now running the widget shows everything properly. Now I just needed to implement the functionality. Luckily for me, I already had code to do the heavy lifting, which I quickly moved from my old Escaper program into this one.

_$, do_escape, and html_escape were easily added, and I didn't have to change them at all. But remember when I said I didn't put a button in? How is the code going to be escaped?

I wanted to do it once per second, automatically, and then whenever something changes in the source box. So I set up a timer that executes once per second and calls do_escape(). That function is:

function everySecond() {
    if (timerID) {
        clearTimeout(timerID);
    }
    
    do_escape();
    
    timerID = setTimeout("everySecond()", 1000);
}

(Note: timerID must be declared globally.) And to the function to handle keypresses:

function keyPressed(event) {
    do_escape();
}

So I have it all in place, but nothing would happen if I were to run it now. The keyPressed function has to be connected to the onkeypress event, and everySecond has to be called for the first time to start the timer.

function setup() {
    // put your setup code here...
    
    _$("source").onkeypress = keyPressed;
    
    //start checking for changes every second
    everySecond();
}

That was easy enough. I ran it in the IDE, and it worked. I went to the Finder and double clicked Escaper.wdgt, and it opened up in Dashboard and worked there, too.

And there you have it, folks. Basically anything you can do with a web application can be done in a Dashboard widget. And web applications can do most of the the things widgets can do (the difference is that widgets can make system calls that normal Javascript can't). Which you write should depend on the application in question. How do you want it displayed? Who's going to use it? Do you want quick access to it, or should it be open and running all the time? Et cetera.

To do:

• Make it resize itself automatically, so there is no empty space at the bottom when nothing has been escaped.
• Make a scrollbar, so that the code doesn't go off the bottom of the widget when it's displayed.

Graffiti

You’ve probably heard that web pages can do a lot more than they used to … but I haven’t really shown you that yet. Simple Javascript applications can be fun and useful, but it’s about time to move on to bigger, better things.

Graffiti is a program I wrote that is designed to be an online chat/forum that allows people to make posts which will be propagated in real-time to everyone who has the page open. It has a Javascript interface that you’re used to by now, a PHP middle layer, and a SQLite database. It communicates with PHP using JSON, which I’ll explain in a later post.

The Idea

Why would you want a realtime chat/forum? Well, what if you’re at work, and the AIM port is blocked? Or you want to have a chat with several people, but not all of them are on the same IM network, or one of them has an old version of the client software that doesn’t support group chat? What if you’re reading a forum, and would never know if there have been new posts made since the last time you loaded the page? In any of these cases, it would be nice to have a program that will get around those problems. Graffiti is one such program.

Adding a new post appends it to the bottom of your page and sends it off to the server. In a standard client-server situation, it would be the server’s responsibility to send that post to every connected user. However, HTTP is limited, in that the server cannot “push” data down to clients. The clients must “pull” it from the server. How can we simulate a “push” when we can only “pull”?

By polling for new content. Each client will sit in a loop and check repeatedly for changes to the state of the program. In the interest of not killing the server, we have the clients checking once per second for new content. If there is no new content, the server will say so. If there is new content, the server will send it. This way, everyone who has the page open will be looking at exactly the same page, regardless of who edits it and when.

The Data

The first thing we’ll do is design the database. There aren’t a lot of demands for our data, so it won’t be too much of a problem. I’m using SQLite as my database, but this is SQL 92 compliant code, so you should be able to use it (or very easily port it) to whatever database you prefer.

Our database will have 3 tables and 3 triggers. The first table is the “entries” table, which holds all the posts that are displayed on the page. It needs a field for the unique id, the author, the content of the post, and the time the post was made. The other two tables are the “instracker” and “deltracker” tables, which are to keep track of the posts that have been made and deleted over the course of the program running. These tables help keep everything sync’d up, by giving the client a place to look when they need to know if anything has changed since the last update.

create table deltracker (id integer primary key, entry integer);
create table entries (id INTEGER PRIMARY KEY, author VARCHAR(255), data TEXT, time DATE);
create table instracker (id integer primary key, entry integer);

To use this code in SQLite, just open up your SQLite command line interface, and type these lines in verbatim. Other databases will obviously have different ways of doing it, so you’re on your own there.

% /usr/local/bin/sqlite graffiti.db

Now let’s put our triggers in. We have one trigger that will put the date into the entries table, so that each new post has an exact date that goes along with it. The other two triggers update the instracker and deltracker tables whenever a post has been added or deleted.

CREATE TRIGGER delete_entries_tracker AFTER DELETE ON entries
BEGIN
        DELETE from instracker where (entry=old.rowid);
        INSERT into deltracker (entry) values (old.rowid);
END;
CREATE TRIGGER insert_entries_time AFTER INSERT ON entries
BEGIN
        UPDATE entries SET time = DATETIME('NOW') WHERE rowid = new.rowid;
END;
CREATE TRIGGER insert_entries_tracker AFTER INSERT ON entries
BEGIN
        INSERT into instracker (entry) values (new.rowid);
END;

You can add these triggers to your SQLite database from the command line.

% /usr/local/bin/sqlite graffiti.db < triggers.sql

Our Javascript code needs to support this database as well. We will define a class, called Entry, that corresponds to the data in the entries table.

function Entry() {
    this.type = "";
    this.id = "";
    this.author = "";
    this.data = "";
    this.time = "";
}

Note that there is a member in our class for each field in the table.

The Interface

Now that our data is all set up and ready, let’s define the interface. It’s not very complicated, but we will have to modify the page at runtime quite a bit.

The page itself doesn’t need much in the way of code. We simply have two div’s, one of which will hold the list of all the posts, and the other will be where the user can add new posts.

<div id="outer"></div>

<div id="add_div">
    <input type="submit" value="Add new" onclick="openAdd();" />
</div>

The CSS to style this program is equally simple. We just define classes for an entry, author, entrypanel, and the bottom. The entrypanel is where the delete button for each post will go, and the bottom is where the content of the post will be displayed.

.entry {
    position:relative;
    border:1px solid black;
}

.author {
    position:relative;
}

.entrypanel {
    position:absolute;
    top: 0px;
    right: 0px;
}

.bottom {
    position:relative;
}

The Functionality

Now that we have the basic interface scoped out a little, what do we want it to do? The button in the add_div will open the Add Post region of the page at the bottom, below all the posts. The user types his name and a message, and clicks the submit button. The post shows up immediately on his page, and is sent off to the server. When someone else posts a response, it will automatically show up at the bottom of the list. If he decides he doesn’t like what he wrote, he can hit the delete button on his post, and it will disappear from everyone’s page.

Communication between the application and the database is enabled by the XMLHttpRequest object. Originally developed by Microsoft and implemented by the other browsers, this allows the browser to request and receive data from a server while the page is loaded, and the browser can do something with said data without refreshing the page. XMLHttpRequest can work synchronously or asynchronously, which means that you can define whether a request should stop the application from running while it waits for data, or if it should continue on its business even if it is waiting for a response from the server. We’re going to use both in this program.

You’ve probably heard the hype already, and I won’t go over it any more. Now … have you actually used the XMLHttpRequest object? Naturally, Microsoft does it a bit differently from everyone else. We’ll use a function that detects whether the browser supports Microsoft’s version or the Mozilla version (Safari and Opera use the Mozilla version), and returns the proper object.

function xmlhttp() {
    var treq;
    if (window.XMLHttpRequest) {
        try {
            treq = new XMLHttpRequest();
        } catch(e) {
            treq = false;
        }
    } else if (window.ActiveXObject) {
        try {
            treq = new ActiveXObject("Microsoft.XMLHTTP");
        } catch(e) {
            treq = false;
        }
    }
    return treq;
}

You can do several things with the XMLHTTP object, but you only actually need a few of them. After you get an object from xmlhttp(), you’ll open it, set its header (needed on some browsers for some types of requests), and send a parameter string. When you open it, you define the action type (GET or POST), the URL to which to send the request, and whether the request should be processed asynchronously or not. If you’re using POST, you have to set the request header, otherwise, you don’t. If you’re using asynchronous, you have to define a callback function. Otherwise, you don’t.

Before I get too far into this and forget to set you up on PHP/SQLite, you need to create a PHP file that can talk to your database. For an application using SQLite and JSON, as we are, the top of your PHP file will look like this:

<?php
require_once('JSON.php');

$json = new JSON();
$file = "graffiti.db";

$db = new SQLiteDatabase($file) or die("Could not open database");

...

unset($db);
?>

This tells PHP to use the object-oriented version of its SQLite adaptor, which we will find very convenient when we need to transfer data. The way JSON works is that it packages up an object, and since our Entry class exactly mirrors our entries table, PHP will see the exact same class structure coming out of the database and the application.

The first thing we need to do when we load the page is load all the entries that already exist in the database. We want this to be a synchronous request, because the program should not run until all the posts have been successfully loaded and put onto the page.

function loadEntries() {
    var req = xmlhttp();
    var param = "load=1";

    //this is synchronous, because we want it to block until all the entries
    //are loaded.
    req.open("POST", "graffitisub.php", false);
    req.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded');
    req.send(param);

    var res = req.responseText;
    entries = JSON.parse(res);
    for (var i=0; i < entries.length; i++) {
        buildEntry(entries[i]);
    }
}

In case you’re wondering what that “load=1” business is, you’re about to find out. That’s the parameter string for the request. If you were loading this in a browser window, that would follow the “?” in the URL. It is also known as a query string. In this case, we’re sending a message to our PHP page that tells it to return an array of Entry objects.

if (!empty($_POST['load'])) {
    $query = "select * from entries";
    $result = $db->query($query) or die("Error in loading");
    $arr = array();
    if ($result->numRows() > 0) {
        while ($obj = $result->fetchObject()) {
            $arr[] = $obj;
        }
        echo $json->encode($arr);
    }
    unset($arr);
}

This sends a query to the database, requesting everything from the entries table. It puts each row in the database into its own object, and puts all those objects into an array, which it returns to the application.

You may also have noticed that for each post that is in the database, it calls a function called “buildEntry”. What that function does is simple: it builds an entry and puts it onto the page. There are two ways to put something onto the page; one way is to use an element’s innerHTML property to simply place HTML into an object and have the browser render it. The other way is to build elements using the DOM’s createElement and appendChild functions. For the buildEntry function, we’ll be using the latter.

We create the div for the entry, then its author span, its entrypanel span, and its bottom div, populate all of them with the proper content, and put it into the page by appending it to the outer div (defined in the HTML).

function buildEntry(entry) {
    var outer = document.getElementById("outer");
    
    var div = document.createElement("DIV");
    div.id = "entrydiv_" + entry.id;
    div.setAttribute("class", "entry");
    
    var top = document.createElement("DIV");
    
    var a_span = document.createElement("SPAN");
    a_span.setAttribute("class", "author");
    var at = document.createTextNode(entry.author);
    var tt = document.createTextNode(entry.time);
    a_span.appendChild(at);
    a_span.appendChild(document.createElement("BR"));
    a_span.appendChild(tt);
    top.appendChild(a_span);
    
    var t_span = document.createElement("SPAN");
    t_span.setAttribute("class", "entrypanel");
    var del = document.createElement("input");
    del.type = "submit";
    del.value = "Delete";
    del.onclick = function() {
        delEntry(entry.id);
    }
    t_span.appendChild(del);
    top.appendChild(t_span);
    
    var bottom = document.createElement("DIV");
    bottom.setAttribute("class", "bottom");
    var d_span = document.createElement("SPAN");
    d_span.innerHTML = entry.data;
    bottom.appendChild(d_span);
    
    mostRecent = entry.id;
    
    div.appendChild(top);
    div.appendChild(bottom);
    outer.appendChild(div);
}

Now that we can load all the pre-existing entries, we need to be able to add our own. The single button isn’t really going to allow that, so we need a function that will expand the add_div div so that new posts can be entered. openAdd() uses the innerHTML property to build its HTML.

function openAdd() {
    var add_div = document.getElementById("add_div");
    var ih = "";
    ih += "<input type='submit' value='Cancel' onclick='closeAdd();' />";
    ih += "<input type='submit' value='Save' onclick='addEntry();' /><br />";
    ih += "Author: <input type='text' id='add_author' /><br />";
    ih += "<textarea id='add_data' cols='100' rows='15'></textarea>";
    add_div.innerHTML = ih;
}

If you can open it, you should be able to close it, so we have a corresponding closeAdd() function.

function closeAdd() {
    var add_div = document.getElementById("add_div");
    add_div.innerHTML = '<input type="submit" value="Add new" onclick="openAdd();" />';
}

As you see above, the openAdd function calls addEntry() when its button is clicked. We now need to write that function. We get the values of the author and data fields in the add_div, and send it off to the PHP page using XMLHttpRequest. For this we want to use asynchronous processing, so that the page doesn’t get slowed down when we click the Save button. This function also updates the global mostRecent variable, which is always the id number of the post most recently added to the page.

function addEntry() {
    var req = xmlhttp();
    var author_e = document.getElementById("add_author");
    var data_e = document.getElementById("add_data");
    
    var e = new Entry();
    e.author = author_e.value;
    e.data = data_e.value;
    var param = "insert=" + JSON.stringify(e);
    req.open("POST", "graffitisub.php", true);
    req.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded');
    req.send(param);
    req.onreadystatechange = function() {
        if (req.readyState == 4) {
            if (req.status == 200) {
                var res = JSON.parse(req.responseText);
                
                buildEntry(res);
                data_e.value = "";
            }
        }
    }
}

Our PHP now has to be updated to support this. We build our query from the passed Entry object, ship it off to the database, and (this part is kind of tricky) send another request to the database to get the entire object that was just inserted. We return this object to the application. The reason we do this is because we want to get the time and id number of the post, which are only known after it has been inserted. Notice above that the addEntry function is designed for this.

if (!empty($_POST['insert'])) {
    $input = $_POST['insert'];
    $value = $json->decode($input);
    $insQuery = "insert into entries (author,data) values (\"".$value->author."\", \"".$value->data."\")";
    $insResult = $db->query($insQuery) or die("Error in query");
    $query = "select * from entries where (id=".$db->lastInsertRowId().")";
    $res = $db->query($query) or die("Error in query 2");
    if ($res->numRows() > 0) {
        $obj = $res->fetchObject();
        $value->id = $obj->id;
        $value->time = $obj->time;
        echo $json->encode($value);
    }
}

Now that we can add posts, we need to be able to delete them. We wouldn’t want our page to get cluttered up with useless crap (especially when we’re testing it out, right?). The delete button on each post calls the delEntry function. This sends a request to delete an entry to the PHP. We use asynchronous processing here because we don’t want to block the rest of the program, but it doesn’t need a callback because nothing is returned from a delete request.

function delEntry(id) {
    var req = xmlhttp();
    var param = "delete=" + id;
    
    req.open("POST", "graffitisub.php", true);
    req.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded');
    req.send(param);
    //don't need to set a callback, because nothing needs to be done after
    //the post has been removed.
    
    removeEntry(id);
}

The PHP code to handle a deletion is probably the simplest thing you’ll have to do. It just takes the id number you sent it from the application and deletes it from the database.

if (!empty($_POST['delete'])) {
    $delQuery = "delete from entries where (id=".$_POST['delete'].")";
    $delResult = $db->query($delQuery) or die("Error in deletion");
}

Once the post has been deleted from the database, we have to remove it from the page. We do this by calling the removeEntry function. This updates the global deletions array (which is supposed to match the deltracker table) and removes the post from the page using the DOM removeChild function.

function removeEntry(id) {
    deletions[deletions.length] = id;
    var outer = document.getElementById("outer");
    var div = document.getElementById("entrydiv_"+id);
    if (div) {
        outer.removeChild(div);
    }
}

Keeping the different clients sync’d with the database and each other is the hardest part of this application. We’ll do it with a check to the database once per second for each client (the timer uses a global timerID variable). We send the most recently added entry id to the database, and it responds with a carefully crafted array. There are always two items in this array. The first is an array of new entries added after the one most recently added by this client (this will often be empty). The second is another array of entry id numbers that have been deleted. This mirrors the deltracker function.

function checkForEntries() {
    var req = xmlhttp();
    
    if (timerID) {
        clearTimeout(timerID);
    }
    
    var param = "last=" + mostRecent;
    req.open("POST", "graffitisub.php", false);
    req.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded');
    req.send(param);
    if (req.responseText) {
        var res = JSON.parse(req.responseText);
        for (var i=0; i < res[0].length; i++) {
            buildEntry(res[0][i]);
        }
        mergeDeletions(res[1]);
    }
    
    timerID = setTimeout("checkForEntries()", 1000);
}

The PHP for this is fairly complicated. It has to create the three arrays I mentioned above, check the instracker table for new entries, get all their id numbers, get those posts from the entries table, and put them into their positions in the arrays. It then needs to return the deltracker array.

if (!empty($_POST['last'])) {
    $last = $_POST['last'];
    $totalArr = array();
    $insArr = array();
    $delArr = array();
    
    $trackQuery = "select * from instracker where (entry=".$last.")";
    $trackResult = $db->query($trackQuery) or die ("Error in tracker");
    if ($trackResult->numRows() > 0) {
        $track = $trackResult->fetchObject();
        $tq2 = "select * from instracker where id>".$track->id;
        $tr2 = $db->query($tq2) or die ("Error in tracker");
        if ($tr2->numRows() > 0) {
            while ($obj = $tr2->fetchObject()) {
                $eq = "select * from entries where id=".$obj->entry;
                $er = $db->query($eq) or die ("Error in tracker");
                if ($er->numRows() > 0) {
                    $o2 = $er->fetchObject();
                    $insArr[] = $o2;
                }
            }
        }
    }
    
    $dq = "select * from deltracker";
    $dr = $db->query($dq) or die("Error in deltracker");
    if ($dr->numRows() > 0) {
        while ($obj = $dr->fetchObject()) {
            $delArr[] = $obj->entry;
        }
    }
    
    $totalArr[] = $insArr;
    $totalArr[] = $delArr;
    echo $json->encode($totalArr);

    unset($totalArr);
    unset($insArr);
    unset($delArr);
}

When it gets the data back from the database, the program now has to make sure everything is sync’d. It loops through the new entries and builds each one. It also needs to take the deltracker array (the second item in the returned array) and merge it with the deletions array (global). The mergeDeletions function simply loops through the given array, starting at the index that would be past the end of the deletions array, and removes the entries specified.

function mergeDeletions(del) {
    for (var j = deletions.length; j < del.length; j++) {
        removeEntry(del[j]);
    }
}

We’re almost done now. The only thing left is to write the init function, which will run when we load the page. It needs to initialize the global deletions array, load the existing posts, and then start checking for new ones.

function init() {
    deletions = new Array();
    loadEntries();
    checkForEntries();
}

I mentioned the three global variables we needed as we went along. They are defined here:

//global variable for the timer
var timerID;

//global variable to track the most recent received id
var mostRecent;

//global deletions array, to keep track of all the deletions made
var deletions;

Recap

If you’ve followed along and built this program, it should now be working for you. Clearly, using XMLHttpRequest and JSON, you can do things with web pages that weren’t possible at all just a few short years ago. This Graffiti program is one such page. Previously, if you wanted to do something like this, you would have had to write a Java applet or a Flash program, both of which require clunky plugins (which I personally hate). Now you can do it with Javascript, which ends up being much nicer for the user.

Beyond the application itself, what this tutorial should have shown you is a kind of development process. The process goes from the IDEA phase, where you sit and dream up what you want your program to do, to the DATA phase, where you define your data structures (including the database), to the INTERFACE phase, where you write your HTML and CSS and decide how the program is going to look, to the FUNCTIONALITY phase, where you write your Javascript (and PHP) and give your program the desired behavior.

To Do

• Make it look nicer. It’s either a forum or a chat, each of which have their own display requirements.
• New posts come in at the bottom of the page. This isn’t always what you want. So update the program so it prepends the new posts (puts them at the top). And make it an option!
• Think about users … what will have to be done to this program to create a login system so that only the author of a post can delete it?
• Dynamically change the timeout interval based on usage to maximize efficiency.

HTML Escaper

Yesterday I posted a tutorial, and the code I put up was displayed in images. I was not satisfied with that solution, because images are not searchable and also you won't be able to copy and paste the code. So I needed to find some way to display my code on this blog. It automatically parses HTML, so if I just put the code up, it would render instead of displaying the code. So I decided to write a small program to escape HTML so it can be displayed on this blog.

It's fairly simple, and probably not robust enough at this point for general use. Basically all it does is change < to &lt;, > to &gt;, tabs to four &nbsp;'s, spaces to &nbsp;, and newlines to <br />. This should be good enough for most code, but if you need it do more things, it's easy enough to update the function.

What I want to do is have a textarea on the page where you can put in your HTML code, and click a button which will escape it. Then it takes the escaped code and gives it to you in a couple of ways. First, obviously, you will need the code in a form that you can copy and paste it into an HTML form and it will display properly (after all, that's the point). To do this, I've made a second textarea that will hold the escaped text. But that escaped code looks like gibberish, really, and it's not easy to tell how it'll look. So this program also renders your text at the bottom of the page, so you can see what it will look like once it's been escaped.

Here's the HTML:

<textarea id='source' rows='20' cols='40'></textarea>
<input type='submit' value='Escape' onclick='do_escape();' />

<textarea id='resultcode' rows='20' cols='40'></textarea>
<div id='result'></div>

(I used the Escaper program to generate that, in case you were wondering if it works. It does not make the border or change the font, I had to do that manually.)

As you can see, the interface code for this program is quite simple, but it works very well. The Javascript is equally straightforward. The do_escape() function calls the html_escape() function once, and puts the same escaped text into both the "result" and "resultcode" fields.

    function do_escape() {
        var esc = html_escape(_$("source").value);
        _$("result").innerHTML = esc;
        _$("resultcode").value = esc;
    }

The html_escape() function does the meat of the work in this program, but is actually very easy to understand, and is even easier to implement. Basically, it takes a string and goes through it character by character. If the character being examined doesn't need to be escaped, it is just passed along. Otherwise, it is replaced by its escaped version. The fully escaped string is returned once it is ready.

    //need to convert the given string into html-escaped form
    function html_escape(string) {
        var rval = "";
        for (var i=0; i < string.length; i++) {
            switch (string[i]) {
                case "<":
                    rval += "&lt;";
                    break;
                case ">":
                    rval += "&gt;";
                    break;
                case "\t":
                    rval += "&nbsp;&nbsp;&nbsp;&nbsp;";
                    break;
                case " ":
                    rval += "&nbsp;";
                    break;
                case "\n":
                    rval += "<br />";
                    break;
                case "&":
                    rval += "&amp;";
                    break;
                default:
                    rval += string[i];
                    break;
            }
        }
        return rval;
    }

You can put this code onto your local server and leave it going in a tab in the background for every time you need it. Next time you want to post code in a forum, you don't have to wonder if the forum will render your code as text or as HTML. If you want it displayed as code, just escape it yourself!

I would have loved to be able to put a demo of it right onto the page, but it seems that that won't work. You'll just have to put it on your own machine and run it from there. Enjoy!

Interest Rate Calculator

A good program to write when you're just starting out should be simple enough to understand quickly yet demonstrate enough features of the language and environment that you will have a base of knowledge to fall back on and build off of when you start writing other, more complicated, programs. One of the first I typically write is an interest rate calculator. It's nice and simple, you use plenty of input, and the formula uses a couple of features of the math libraries, so you can see what the language can do.

The first thing to do when designing a new program (after you've decided what program to write, which we've just done) is to design the interface. Many times, if you can design a good interface to a program, which should work in an obvious way, the features you need to implement become obvious and the code pretty much falls out of the interface. (That's why Cocoa programs are made by designing your interface, connecting actions and events, and then writing code to handle the interaction with the interface.)

To calculate interest, you need the initial amount, the interest rate, and the time passed. We'll need a text field for each of those, another text field to show the result, and a button to initiate the calculation. Simple enough, I'd say. I usually don't like to do this, but this works pretty well with a table layout. So here's the HTML code for the interface:



If you type that code into your favorite editor, and save it into an .html file, you'll be able to load it in your web browser. It looks exactly the same as your finished program will. So now it's time to implement the functionality!

To save time and code space, I typically include a shortcut function in my projects that allows me to type less when I need to get a reference to an element on the page. "document.getElementById()" will get a reference to an element, but it's an awful lot of typing if you're going to need to do it a lot. Here's my time-saving function:



It's probably a good idea to put this function in a separate .js file that isn't specific to any one project. That way, you can easily take all your personal utility functions along at the beginning of a new project. And if you're doing things right, you'll start to amass a few things in your lib.js file.

Now that we have that in place, we should write the calculate() function. It needs to get the values of the initial amount, the interest rate, and the number of years, then it needs to use those values to calculate the interest earned, and put that final value into the "Total" field.

The formula for calculating interest, compouded yearly: TOTAL = INITIAL * (1 + RATE) ^ YEARS

Once we have that formula, the function is trivial:



The first three lines get the values of the input fields and parse them into numeric values. This ensures that it won't try to calculate a total dollar value if there is, for example, a letter or some other non-numeric character thrown into the works. The initial dollar amount and the interest rate are decimal values, so we use parseFloat(), but the number of years must be a whole number, so here we use parseInt(). Since we want the interest rate to be entered in a human-friendly way (5% interest, rather than 0.05), we will divide the rate value by 100.

Then we calculate the total value based on our formula, and place it into the "total" text field. This is the output of our program, and it immediately shows up on the page. No page refreshes here.



Notice that there are only two decimal places in the total field. This is exactly what we want, of course, because monetary values are displayed with only two decimal places. In order to make sure there are no more than that, we use Javascript's Math.round() function to round the value, multiplied by 100, down to the nearest integer, then divide it by 100 to get the decimals back. (For future reference, this technique works for any number of decimals, just change the magnitude of the multiplier/divisor, which in this case is 100.)

The Math.pow(x,y) function does about what you'd expect it to do: it returns x^y.

Now that program was simple enough, right? And it doesn't really scratch the surface of what you can do in a modern web application. Here are some things to try out on your own, to improve the interest calculator program:

• Give it some style! Right now, it's a fairly ugly program. It uses the system default font and font size. Create a CSS file and get to work on making it a bit prettier.
• Have you noticed what happens when the number of cents in the total field is divisible by ten? And what if there are no cents at all? Try to update the calculate() function so that it will always give the proper number of trailing zeroes.

Now this was a fairly simple little program, but I hope it helped get your feet wet. The next one will be a bit more complicated and interesting. It's coming soon, so don't go far!

Introduction

Hi there. I'm Sean, and I do some programming. A lot of the things I learned have been from the web, and many of those were in some random guy's blog about some cool/stupid thing they did. So I've decided to be one of those random internet people, and help out people new to programming with some of the cool/stupid things I've done.

But on thinking about it, I don't really consider much I do to be particularly cool, and it's not even that stupid. So why not dedicate to something else? I have decided to make this not so much a showcase for things I've done as a place for people to learn. I'll try to publish reasonable tutorials on various things ... for the most part they will be on modern web applications (or AJAX, if you're into the term), as that's what I've been spending most of my time on lately.

And on my larger projects, I'll try to post updates on my progress, just so that anybody who finds themselves interested in them (for whatever reason) can follow along and make fun of my slow rate of progress or lack of glitzy features. Or something.

The first real post should be up shortly. Don't hold your breath (it's not good for you).