elarson’s posterous

I read this article by Lily Allen on music piracy. While I'd like to think that music should be free, after looking at a few offers we've received from labels it is difficult to deny that music piracy has been considered part of the equation.

The big issue is not that record labels are struggling. This is a good thing. Labels have been loan sharks for entirely too long so good riddance. But, as crappy as big labels can be, there are a wealth of smaller labels that do aim to release great music to people because they appreciate what artists are doing. These range from friends that have friends in bands they want to help out, all the way up to labels who have established themselves as viable businesses. For bands, often times a label is the sole party that really can break a band.

There are are plenty of other ways a band can break, but the most common is through a label. The label puts out a record, pays for press, helps improve the bands stature and spreads the word with the result being the band finds a fanbase. The most important thing in this whole scenario for a band is finding the fans. While the critics can be nice to sway and Carling massive attention on blogs has its perks, the real deal is to get your music to fans. As a band, the closer you get to your fans and the more direct relation you build, the better the payout.

In the past bands could make money selling records. It was a big lottery, but it was still possible. Bands also were given the responsibility to do with their money as they chose. A band gets a big advance from the label and uses the funds to build their business (or buy drugs). Now, labels don't make nearly as much money selling music, so they are looking for other revenue streams. This means "partnering" up with an artist. The negative term has become the 360 deal where a label effectively can act as a manager, taking a percentage of the gross income. The more common occurrence from what I've seen is that labels aim to find targeted income areas to cover costs through the artist.

This in and of itself isn't too bad. It can be a huge help keeping the band funds liquid. If a band lands a huge tour or gets massive success, every t-shirt size they don't have is money lost. If a label can make sure you have enough shirts, hoodies and vinyl, that is a definite benefit. Labels can also help find placements and synchs for songs, which can be very lucrative. Again, very nice.

The problem is that while these benefits can be helpful, they can also preclude you from building your own business. If your label is your merchandising outlet, you probably can't make t-shirts yourself. If you have a three record deal, that is probably 3-5 years lost building part of your business. You won't be able to build up your own relationships with another manufacturer or put together your own fulfillment facilities. Likewise, if your label owns some of your publishing, that can end up being a lot of money. Say you have one song that gets 5 placements a year. If you average $2k per placement that is $10k. Over 20 years, that is $200K, which seems pretty good. Take 25% and give it to the label, pay another 20% in taxes and split it up between band members and it quickly becomes rather depressing. Couple that with the fact some labels are asking for actual percentages of tour revenue as well. Also realize that the band is most likely having to pay a lawyer, manager and booking agent (and in some cases a tour manager). Many of these support staffs used to take flat rates, but now all focus on percentages.

Before, if a band got 12% of a record sale they were doing pretty well. They could go on the road and make good money. Now, you don't get any money from record sales even though your percentage might be 50%. In addition to that, you are effectively taxed by your "team" on everything else you do. 5% for your lawyer, 10-15% for your manager, 10-15% for your booking agent, 10-20% for a tour manager, and 10-15% from your label. So, between 45% and 70% of the money you make disappears before you see it. Even if you do "break" and get a decent following, getting $5k a show doesn't mean much over the course of a year. In the famous words of Bill Cosby, "You have eaten yet!" Even if you take home half of the $5k for 30 dates ($2500 * 30 = $75k) you've yet to pay for gas, food, lodging, equipment and taxes. When you consider this money has to cover all the band members living expenses both on the road and off, things start to get rather tight. This doesn't even consider off road maintenance such as wardrobes and props (fog machines and can lighting is not cheap).

I honestly have to agree with Lily that music piracy doesn't really help the artists. It does help destroy the old system, which I do think is a good thing. But, unless there are some provable models for being an artist that don't involve taxes by everyone that helps you break, it doesn't seem like it is any better for artists. In a sense it is potentially worse because at least before, labels sold a product. The initial investment or R&D were directly tied to selling music. Now, everyone wants to get shares in the artists career.

The music business has been compared to the tech industry and venture capital, but I don't think that comparison is quite right. Music doesn't scale the same way technology does. Craigslist is a great example where a really small group of people have been able scale a small business from one city to a whole country. A band simply can't all of a sudden put out an album a day while touring non-stop. They need others to help keep the message out there and no one works for free. The result is that in order to break and make enough money to be reasonably stable, the actual revenue needs to be pretty huge.

With all this said, the fact still remains that I love playing music. It is an honor that people want to invest money in the music that I help create. While we plan on trying to be as shrewd as possible in terms of music as a business, the performers and artists inside just want to get our music out to as many people as possible. It is really all about fans and making that connection, so if that means giving up some ownership in order to help get our name out there, I'm kind of OK with that. The only requirement is that if you want a share of our music, you need to really earn it and be a fan.

Loading mentions Retweet

It feels as though web development has begun to focus on other parts of the stack. Up until recently, the framework decisions seem to be the biggest focus, with MVC based patterns reigning in the masses. At this point though, there is a wealth of documentation and options that make trying out the latest and greatest MVC framework slightly passe. To combat this stagnation in hip web technology, the focus has changed slightly to server technology.

Here are few recent articles to make point regarding the state of web development:

• Unicorn is Unix
• Python is Unix
• Announcing Tornado
• Announcing Diesel

While most of these ideas and concepts are not relatively new in the realm of computer science, they are new to me. I represent a rather large audience of web developers who did not necessarily see web develop from socket libraries to Rails. Instead, my experience began with PHP and didn't include any understanding of what actually happens on the web. Fortunately though, my own forays into web development exposed some consistent patterns that helped me understand what was really happening. So, with these new-ish ideas coming around, it seemed like a great learning opportunity to become better acquainted with more the lower level aspects of web development.

While I have a few projects laying around my home directory for each of the ideas mentioned above, this one is specific to Diesel. I'm somewhat partial to for social reasons. When thinking of an idea for something that might very well benefit from being asynchronous (that is not a chat server), I came up with the idea for a database hub.

The idea for Repeater is that you set up a proxy that will "repeat" the requests to a set of services. For example, if you were using CouchDB, you could set up Repeater, which would be a round robin proxy for the different instances and allow updating all the instances with one request. This doesn't work yet! But that is the idea.

What does work is a basic proxy that will balance between some set of services. It works pretty well from the stand point of basic tests not failing, but I have no idea if it would be extremely slow in practice. The thing that got me stuck is how to handle more requests without blocking when requesting the other site. From the code, you can see where I started messing around with threads to make sure I don't block. The gotcha in all this was that I couldn't figure out a way to effectively yield a sleep or join where the join is when the thread has finished making the request. I'm sure there is a way to do it, but my experiments were not very fruitful.

My overall impressions are pretty good. I'm still a bit hazy on the applications where an async system is an order of magnitude better than a threaded or forking (or both) system. The chat example seems obvious because you have a direct relationship between a client connection and a single process that needs to make responses in order. This is similar to a database connection in that it may need to handle a large amount connections, but at the same time, as soon as you start doing things like requests over the web, it seems inevitable that blocking will be an issue. Although, it seems very solvable.

Loading mentions Retweet

We recently returned from a tour. While on the road, I'm still working. I have a slick bean bag chair, curtains and MiFi router, which makes it possible to get my work done while cruising down the freeway. This is something I've discussed before.

My two recent annoyances with my setup are not having an automatic way of completing an email address and slow updates to the server. So far, the slow updates will probably involve setting up a local IMAP server to sync to and from. It is something that I've started looking into, but haven't spent that much time on it. It ends up being easy enough to just ignore email and save the time. What is rather annoying is having to enter email addresses.

It is not that bad, but it is relatively time consuming. My typing could be much better and it can be easy to make mistakes. Also, there are many times I'm copying the email address from some other resource such as a web page or IRC, in which case, there is a lot of buffer management to deal with. Again, not a huge deal but enough to warrant looking into solutions.

Fortunately, like most things in the Emacs landscape, there was already a good solution in place. The BBDB or the Insidious Big Brother Database. I initially tried using the tips mentioned on emacs-fu but found it didn't work. Fortunately, I found the solution was incredibly convenient and directly in the Wanderlust docs. The result, is now I'm collecting email addresses for completion much like I would get in gmail in addition to having an actual addressbook I might consider using. Pretty nice ROI for googling a bit.

This is realization you need a feature is a pretty common occurance in Linux. The conceptual basis is usually there to create a solution, but often times it takes a bit of work to really get things configured. Emacs is very similar, with the exception being it is almost expected that the configuration might very well be writing a new mode or piece of functionality yourself. It serves as a good reminder that I shouldn't go recommending Emacs to my parents anytime soon. Although it would be pretty cool if I did get a call from my dad asking about setting up a keybinding for some lisp function he wrote.

Loading mentions Retweet

The other day I got rather caught up in the whole * is Unix discussion and wanted to get a better understanding of some of the more low level details associated with writing servers. Eventually, I had a crude version of CherryPy that forked instead of used threads. The benchmark in CherryPy seemed to suggest it was a great deal faster, but honestly I think that was a fluke, especially considering I'm pretty sure there are some serious deficiencies in terms of process management. When Bob posted his forking example on the CherryPy list and mentioned signals, it was clear that I was missing some pretty important understanding.

One thing I thought seemed helpful was that it seemed to use both processors on my system. My understanding is that calling fork will create two separate processes, which effectively should place at least some of the forked processes on different processors. My understanding is that this is how the multiprocessing module works (on Linux at least). With this new cursory knowledge under my belt it was interesting to see an article in my ACM Communications magazine remark on the future of computing and the need to handle multiple processors in a similar manner as sequential paradigms. I should note that this is one of the very few times there was anything remotely interesting to me in my ACM magazine, so I felt compelled to check it out. The idea is not new to me and while the arguments that this is a huge issue is pretty valid, I also started thinking that maybe it is not as big a deal in practice.

The reason for this is the web. Back in the early days of computing people had dumb terminals and logged into a server where all the actual "computing" was performed. While thick clients have been the rage for quite a while, the web is effectively becoming the mainframe in the sky in terms of where people are doing their computing. This should be very helpful in making the jump to parallel computing because there s a definite history and set of tools that have been developed to scale that allow parallel processors. In other words, who really cares if your computer has 2 quad core processors when your searching email online, browsing facebook, checking your bank statements and reading the news through your web browser.

On the web, what really matters is that the sites doing the processing have optimized their architectures to handle the load. The other side of the equation is the browser, but in reality, this is an area that is already becoming more robust. Google Chrome is a good example with its separate processes for each tab, but I would even argue that Javascript is well suited to handle distributed tasks. We've seen a ton of articles on using async servers. Javascript is already using an async model that is only getting faster with the recent developments in new Javascript engines. None of this means that parallel computing and utilization of more processors is not very important. I just don't think it is quite as critical as some might think. That doesn't mean programmers shouldn't try to understand it. After all, it is a hard problems and programmers typically like hard problems. My whole point though, is that the problem might be more of a fad than an actual crux for the IT industry.

Loading mentions Retweet

Some folks are tired of ORMs and raise a good point regarding exposing SQL generation in ORM libs. Most of the time when I use an ORM, there is a moment where I wish I could just get on with things. The declarative approach for creating tables usually is one hurdle that annoys me. Another is defining relations, since it seems so trivial to do that sort of thing in the query. At the same time, I'm far from a heavy database user and can imagine my "it's so simple" mindset quickly deteriorating to massive complexity without much work. Though, I always did like the web.py db layer. It pretty much gave you queries and returned the results in objects. Pretty simple stuff. I think it also let you update those objects and that would be persisted in the DB, but I could be wrong about that. At one time I tried to use it on its own but didn't have much luck. One gotcha is that the db layer depended on running within web.py and had expectations for keeping a threadpool and the like. Thinking back, it could have been very helpful if web.py's db layer did provide some helpers for writing SQL. Based on the article and the comments, it sounds like SQLAlchemy is already able to expose the query mechanism. I'm not sure how this works within the scope of providing model objects, but it sounds pretty promising nonetheless. Likewise, Dejavu, a personal favorite excels an improving queries via translation from language constructs. The NoSQL movement is partially exciting because instead of thinking about relational algebra you think about looping and applying functions. Dejavu uses the same ideas and exposes the Python to SQL translation with its Geniusql lib. Even though I don't really use databases very often, it is interesting to see how people are handling the progressive issues of using an ORM in the real world. Just as databases aren't dead, I believe ORMs are still very important and will most likely become even more important. There is a ton of well tested boilerplate code in ORMs that make creating models much easier. My guess is that even in a world full of NoSQL, we'll still see ORMs help out when it comes to keeping tabs on domain objects. What we won't see is one ORM that supports every database or persistence layer known to man, although there will be people who try to write it.

Loading mentions Retweet

Today I had the desire to write something down, but really didn't have a concise idea of what to write about. So this post is just going to be a small summary of some thoughts and experiences.

Free Software and Open Source

I recently read the RMS opinion on Codeplex and Miguel's response. After a quick glance over at planet gnome I noticed a few people taking sides and it occurred to me that the whole argument is rather silly. When I was in college the concept of free software made a ton of sense. Looking back it was because I didn't have any money, so generally anything free made a ton of sense. Now that I'm a full fledged tax paying adult, the glamour of free software has lost its glitz. It is not that free software has become unimportant or useless. What has happened, in my mind at least, is the arguments associated with free software have become rather stale. By stale, I simply mean it isn't anything exciting for me personally. I think free software is critical, but I have better things to do than care about it in its own right. I'm probably just getting old, but it was an interesting realization for me nonetheless.

Test Driven Development

At work I've been trying to improve my tests. By "improve" I really mean write them in earnest. It is a really difficult thing to write code using TDD. It is a similar approach to modeling in that it forces you to consider an abstract idea of what some code should do and look like. TDD is sort of like UML in the age of Ruby on Rails, which is kind of funny as the recent web frameworks and NoSQL all suggest rapid prototyping over planning before coding. While both UML and TDD are doing pretty much the same in terms of hashing out code, the obvious benefit of TDD is that you get something that can be used in the future. At the same time, a well tested code base is not that important if the tests are bad and are hard to run. Testing in web browsers is the most obvious case in point. The larger point then is obviously that planning, whether through tests, visio or some hodge podge of tools, is helpful for writing better code. It might also be argued that it is faster since the design is fleshed out to some extent, but I would ask if the time spent planning is included in that calculation and if it is a real calculation at all. Programmers have a nasty habit of estimating because of the constant requirement to create hypotheses in debugging. My bet is that many of the virtues of TDD (like UML as well) are overblown and the only real benefit is forcing a developer to focus on what the problem is. One of my issues is that it creates a whole new class of code that deals with testing. This is totally fine, but where are tests for the tests! It seems like a story that we'll probably never see the end of.

Text Browsing

I'm going to suggest that if you're a programmer, it would treat you well to try out a good text web browser. My recommendation is w3m due to its Emacs integration, but anything that can keep you in your work environment works. My guess is vimmers would get similar usage from links/lynx assuming the terminal is their environment. The reason being is that if you are constantly editing text and reading it in your dev environment, browsing the web textually can be a helpful tool to keep focus. For me, I get the same keybindings, easy copy and pasting, and simpler window/frame/buffer management. Beyond this though, it feels faster when it comes to reading documentation and finding helpful code. Your milage may vary, but it sure couldn't hurt to try.

Administering Systems

At work we recently rolled out a new system. It isn't actually new, but is in fact the latest step in an improvement to a current system. What always strikes me about the smart folks I work with is how gracefully they walk the line between system administrator and programmer. The two fields are completely intertwined, but the best programmers are those that have the better understanding of both sides. This is probably partly why I'm not that great of a programmer! For whatever reason, my mind doesn't ever seem to really indulge in the system administration side of things. It is always a challenge for me to make pre-existing software work the way I think it should. That doesn't mean I'm not trying of course! But it does mean that I have a ton to learn and will for the foreseeable future.

ACL Wrap Up

This past weekend was ACL in Austin and it was a blast. We saw Them Crooked Vultures, The Walkmen, School of Seven Bells, Broken Social Scene and some guys from Phoenix DJ. We also played a show with The Riverboat Gamblers and The Soldier Thread. It was a ton of fun. On the road we don't really get to hang out that much. There is usually somewhere to drive, something to load or unload or something to sell that usually keeps us busy. It was great to come home, rest and then have a great weekend of music and friends. We didn't go to ACL proper and I'm glad. There were plenty of bands I would have liked to catch but the weather was horrid and my guess is I would have been pretty miserable in the mud. Hopefully next year there will be some nicer weather. Who knows, maybe we'll even get to play!

Loading mentions Retweet

It has taken a long time, but I've finally managed to get email working relatively well in Emacs! In the end Wanderlust was the client of choice for me. It has a pretty simple file format for organizing mail and the keybindings have already begun to be rather natural. The reasoning behind my quest for merging my email and emacs had more todo with my recent tour than an actual desire or need for checking mail in my text editor. I was perfectly happy with Gmail and in fact I'm technically still using it through IMAP. The real reason was that I wanted to try and reduce my bandwidth usage. On tour I had a 5 Gig cap on my network usage. Having never really monitored this sort of thing before, I had no idea how much bandwidth things like IRC might use when you're talking about every day, all day usage. To make things worse, I forgot to grab the USB cable that would let me check my current usage for the month, so I was essentially flying blind!

My fear of an enormous bill made me consider what I could do reduce my bandwidth. The obvious avoidance of listening to music and watching videos was easy. Also, reading local versions of docs was another helpful tactic. This lead me to see a few things about w3m, which I had tried to use before in Emacs. It dawned on me that things like code examples viewed in a text based web browser makes nothing but sense in my text editor. It makes things like copying and pasting code samples a breeze. After getting used to w3m in Emacs it became clear that I should really try to finally tackle email. As I said, Gmail has been just fine, so the motivation wasn't very strong to move. The killer feature in my mind was tying things into my todo list and avoiding Gmail's constant stream of Ajax requests. I'm sure the bandwidth savings are practically nothing, but nonetheless, it seemed like if I got things working it might really be helpful in ways similar to w3m.

Once things were configured and I managed to get everything working, I did notice some helpful bits. Silly annoyances that really should never have been a problem smoothed out. I'm talking about basic copy and paste issues I noticed here and there when working with Emacs and the rest of my desktop. Little things like working with Trac was also a good deal faster, although it was something of a let down that I couldn't submit updates to tickets. Finally, it seems more reasonable to keep up with mailing lists as most of the necessary content (code) is close at hand. This last aspect seems to be the biggest draw of using Emacs. I primarily write code and keeping more tools inside Emacs makes it faster to transition between those tools and the code I'm working on. Of course it does seem kind of cool in a geeky way to work almost solely in text, but the main nicety is actually being more productive. I'm hoping in the next couple weeks things can continue to become a little more streamlined. The killer app that I would like to find a good Emacs replacement for is my feed reader. Google Reader is great but something more bare might help to move through things faster and generally filter out the cruft. That said, blogs were one of the first things I cut out in my quest to lower bandwidth. My desire to check what is new in the blogosphere has greatly diminished since taking a break. I'd like to potentially keep it that way for a while.

Loading mentions Retweet

Google is often considered a benchmark in the tech industry for some epitome of a field. It is pretty clear that Google is at the top of their game when it comes to search. One of the technical reasons for this status is due to their infrastructure. Much has been said about how they are able to use standard PC hardware and Linux to create highly distributed systems. The other day it occurred to me that someone should really do the same with food.

When we were driving up to San Francisco, we went by a slaughter house. It occurred to me that it must take quite a bit of fuel to move around so many animals. It was also a pretty disgusting place. It used up a ton of space and generally seemed like an unhealthy place to be producing massive amounts of food. When we saw Food Inc., there was an organic natural farmer that had a wealth of crops and livestock for sale it seemed really efficient. Normally, people look for ways to build bigger and bigger assembly lines. After all, that is what the food industry really is, a huge assembly line for growing, processing and packaging food.

The problem is that you can really only scale one large system so far. At some point you'll have to consider the implications of using a single system that has many single points of failure. For example, when gas prices rise, it effects almost every factor of food production on a number of levels. If farmers could more easily raise their own livestock using local food sources and do the slaughtering and packaging on site, you end up cutting out massive amounts of fuel shipping thousands of pounds of food.

It seems like someone should do something similar using local smaller farms. A distributed network of self sustaining farms could very easily help reduce the costs of transportation and improve the quality of the food. There would be a different kind of problem organizing these small farms, but the issues, while complex, would be orthogonal to problems of creating food at a large scale. This is the same problem companies like Wal-Mart and Target have had to deal with, so applying similar techniques to food seems not only plausible, but very profitable.

The big unknown is how the government could facilitate this movement. It already is primarily responsible for the current state of the food industry because of its subsidies and attention to storable food. Smaller farmers can make the cycle of production simpler which means if one farmer fails or there is a small shortage, you are dealing with a smaller scope that is most likey limited and easy to contain. With this in mind perishable foods can be given attention and small farmers would have a means to participating in the market as a whole. The big necessities from government would be to try and improve the patent situation for seeds, fix the subsidy laws that fund the major food corporations flawed practices and make an effort to reward those farmers who are able to be profitable without relying on government subsidies.

I'm not an expert on this stuff, but it seems like the principles might very well apply. My guess is that as organic foods continue to become more important, companies like Wal-Mart will have to find a way to give customers healthy food on a large scale and faciliate the creation of orginizational tools necessary to feed a nation on the spoils of many small distrubuted farms.

Loading mentions Retweet

I went ahead and started working out how to get my todo list online. I started off pretty simple and ended up with a relatively nice system. The basic idea is that I can push my org files to my webserver and edit them. Likewise, I can pull from the server. It started with some simple paver scripts that uploaded the files and quickly became an actual application.

Here is the paver file for some of the operations:

import os
from mercurial import commands, ui, hg
from paver.easy import *
import subprocess

IONROCK_HG = 'ssh://eric@ionrock.org/path/to/todos/'
REMOTE_TODO = IONROCK_HG # '/local/dev/path/to/todos'

@task
def server():
    import cherrypy
    cherrypy.tree.graft(TodoServer(base_url='/'), '/')
    cherrypy.quickstart()

@task
def create_repo():
    cmd = subprocess.call("fab create_repo:hosts='ionrock.org'", shell=True)

@task
def commit():
    conf = ui.ui()    
    user = conf.username()
    repo = hg.repository(conf, '.')
    files = [f for f in os.listdir('.') if f.endswith('.org')]
    commands.add(conf, repo, *files)
    commands.commit(conf, repo, addremove=True, message='Syncing org files')
    commands.push(conf, repo, REMOTE_TODO)

@task
@needs('commit')
def pull():
    conf = ui.ui()    
    user = conf.username()
    repo = hg.repository(conf, '.')
    commands.pull(conf, repo, REMOTE_TODO)
    commands.update(conf, repo)

@task
@needs('commit')
def update():
    cmd = subprocess.call("fab update_todos:hosts='ionrock.org'", shell=True)
    

The server task was for starting the eventual web application for development. The commit task just automatically commits the current org files and pushes them to the remote server. The pull command does the commit first, then pulls from the remote server. These two commands uses the mercurial libraries to work with the mercurial repos.

The create_repo was just a simple task to create an mercurial repo. More interesting is the update task which updates the remote todo mercurial repo. I'm using fabric for this aspect. It was all really easy. Here is the fabfile:

from fabric import run

def update_todos():
    run('cd /home/eric/htdocs/todo && hg up')

def create_repo():
    run('cd /home/eric/htdocs/todo && hg init')

Hopefully it is really clear what is happening here. Fabric lets you run commands via ssh on a remote server.

The actual todo server is a bit longer but also pretty simple.

import os
import re
import posixpath as path
import difflib
from selector import Selector
from webob import Response, Request
from webob.exc import *
from mercurial import commands, ui, hg
import datetime


class TodoFile(object):
    def __init__(self, fn):
        self.fn = fn
        self.html_diff = difflib.HtmlDiff()
        self.diff = difflib.Differ()
        self.matcher = difflib.SequenceMatcher()
        lines = [l for l in open(fn, 'r')]
        self.matcher.set_seq2(lines)

    def _hg(self):
        conf = ui.ui()
        user = conf.username()
        repo = hg.repository(os.path.dirname(self.fn))
        return conf, repo, user

    def __str__(self):
        return ''.join(self.read())
    
    def read(self):
        return [l for l in open(self.fn, 'r')]

    def write(self, new):
        f = open(self.fn, 'w')
        clean = re.sub('\r', '', new)
        f.write(new)
        f.close()
        conf, repo, user = self._hg()
        date = datetime.datetime.now().strftime('%m-%d-%y %H:%M')
        commands.commit(conf, repo, message='Web write on %s' % date)
        
    def is_different(self, new):
        self.matcher.set_seqs(new.split('\n'), self.read())

    def diff_txt(self, new):
        return list(difflib.context_diff(new.split('\n'), self.read()))

    def diff_html(self, new):
        return self.html_diff.make_file(self.read(), new.split('\n'))        
    

class TodoStore(object):
    def __init__(self, directory):
        self.dir = os.path.abspath(directory)

    def get_todo(self, name):
        for fn in os.listdir(self.dir):
            if fn.endswith('.org') and (fn[:-4] == name):
                return TodoFile(os.path.join(self.dir, fn))
        return false

    def all(self):
        return [fn[:-4] for fn in os.listdir(self.dir) if fn.endswith('.org')]


class Auth(object):
    def __init__(self, creds, login_url, success_url=None):
        self.login_url = login_url
        self.success_url = success_url
        self.creds = creds

    def __call__(self, f):
        def func(env, sr):
            sess = env['beaker.session']
            if sess.get('auth.user'):
                return f(env, sr)
            req = Request(env)
            sess['auth.after_login_url'] = req.url
            sess.save()
            return HTTPSeeOther(location=self.login_url)(env, sr)
        return func

    def login(self, env, sr):
        res = Response()
        sess = env['beaker.session']
        flash = sess.get('flash', '')
        if flash:
            sess['flash'] = ''
            sess.save()
        res.write('''<div>%s</div>
        <form action="%s" method="post">
          <label for="username">Username</label>
          <input type="text" name="username" value=""><br />
          <label for="password">Password</label>
          <input type="password" name="password" value=""><br />
          <input type="submit" value="login" />
        </form>''' % (flash, self.login_url))
        return res(env, sr)

    def handle_login(self, env, sr):
        req = Request(env)
        post = req.POST
        sess = env['beaker.session']        
        if post.get('username') and post.get('password'):
            if self.creds.get(post['username']):
                if self.creds[post['username']] == post['password']:
                    sess['auth.user'] = post['username']
                    url = sess.get('auth.after_login_url', self.success_url)
                    sess.save()
                    return HTTPSeeOther(location=url)(env, sr)
        sess['flash'] = 'Error logging in.'
        sess.save()
        return HTTPSeeOther(location=self.login_url)(env, sr)
            

class TodoServer(object):

    def __init__(self, **config):
        self.conf = {
            'todo_dir': os.path.dirname(os.path.abspath(__file__)),
        }
        self.conf.update(config or {})

        self.auth = Auth(self.conf.get('creds', {}),
                         self.url('login'),
                         self.url())
        

        self.store = TodoStore(self.conf['todo_dir'])

        self.router = Selector([
            ('[/]', {'GET': self.listing}),
            ('/login[/]', {
                'GET': self.auth.login,
                'POST': self.auth.handle_login
            }),
            ('/{name}/edit[/]', {
                'GET': self.edit,
                'POST':  self.auth(self.update)
            }),
            ('/{name}[/]', {'GET': self.read}),
        ])

    def url(self, extras=None):
        extras = extras or ''
        if isinstance(extras, list):
            extras = '/'.join(extras)
        return path.join(self.conf['base_url'], extras)

    def _header(self):
        return '''<html><head>
        <title>org todo server</title>
        <style type="text/css">
        body {
            font-size: 2em; font-family: sans-serif;
        }
        </style>
        '''

    def _footer(self):
        return '''</body></html>'''

    def edit(self, env, sr):
        res = Response()
        req = Request(env)
        name = req.urlvars['name']
        td = self.store.get_todo(name)

        res.write(self._header())
        res.write('''
        <form action="%s" method="post">
        <input type="submit" name="submit" value="save" /><br />        
        <textarea rows="50" cols="80" name="new_body">%s</textarea>
        </form>
        ''' % (self.url('%s/edit' % name), str(td)))
        res.write(self._footer())
        
        return res(env, sr)

    def update(self, env, sr):
        req = Request(env)
        name = req.urlvars['name']
        new_body = req.POST['new_body']
        todo = self.store.get_todo(name)
        todo.write(new_body.strip())
        location = self.url('%s' % name)
        return HTTPSeeOther(location=location)(env, sr)

    def read(self, env, sr):
        res = Response()
        req = Request(env)
        name = req.urlvars['name']

        res.write(self._header())
        res.write('''
        Home | Edit
        <hr />
        <pre>''' % (self.url(), self.url('%s/edit' % name)))
        td = self.store.get_todo(name)
        res.write(str(td))
        res.write('</pre>')
        res.write(self._footer())
        
        return res(env, sr)

    def listing(self, env, sr):
        res = Response()

        res.write(self._header())
        res.write('<ul>\n')
        for f in self.store.all():
            res.write('<li>%s</li>\n' % (self.url(f), f))
        res.write('</ul>\n')
        res.write(self._footer())
        
        return res(env, sr)

    def __call__(self, env, sr):
        return self.router(env, sr)

This is a WSGI app simply because I'm using WSGI for my main application. I save a bit of memory by running all my smaller apps via one WSGI server (CherryPy), which makes a difference as I use a VPS.

One observation I made is that things would have been simpler had I been able to use CherryPy. Things like sessions, form processing and even URL routing would have been built in and made the whole thing a lot simpler in terms of dependencies and actual code.

This also made me realize what the problem is building applications with WSGI. You really need a framework. I don't mean Pylons, web.py or some other WSGI framework. But you will undoubtedly write some glue code to help handle things like request and response objects that help to deal with form handling, sessions and cookies. It is nice to know that it is so easy to create these micro frameworks, but at the same time, it is clear that people would be making bad decisions. I only say that because I'm one of them.

When I think of the micro frameworks I've written throughout the past few years, it is clear that I've had to experiment quite a bit. WebOb was a helpful library for sure, but the API you build translating the request to a WebOb Request means breaking WSGI at some level. That means that you've lost the advantages of WSGI as an API for your application. In my mind, it makes me wonder why then the app was written with WSGI in the first place as there is a solid and proven API already built with something like CherryPy.

I doubt I'll rewrite my whole site anytime soon, but if I do, the application framework will most definitely revolve around the framework rather than WSGI. The advantages that I believed were present ended up being much less than I thought. Having a tool like CherryPy manages to take care of the generic aspects enough while letting me use more opinionated aspects such as templating or databases. You could most certainly substitute your framework of choice, but for me CherryPy is making more and more sense.

Loading mentions Retweet