Jilles’ weblog

On April 12th I presented at PFCongrez, a yearly gathering of PHPFreakz. During the day three other presentations were given. The first one was by Peter-Paul Koch (ppk for short) who presented about unobtrusive javascript. The presentation was given with a lot of energy, enjoyable!

After that, I presented a sneak peak at Marktplaats during which I gave some insight into what it takes to run one of the biggest sites of the Netherlands. It goes into some of the high level production setup, highlights some of the challenges of operating hundreds of database deployments and goes into some of the aspects Marktplaats runs into while using PHP.

The slides are embedded below, or up for download here. Or join the after party discussion.

| View | Upload your own

The other presentations during the day that can be found online:

• Peter Paul Kock, Unobstrusive Javascript
• Stefan Koopmanschap, Symfony

Next to that, I am trying to gauge any interest in our tool to manage database schema’s: DBC. It keeps database schema’s in synch with your application, and allows developers to branch off the database schema much as a version control system allows you to do with your code. If you are interested in this tool, please contact me at “jilles &at& marktplaats . nl”. We are looking to see if there is enough interest to open source the tool.

I have lost my copy of “Godel, Escher, Bach” by Douglas Hofstadter, so I bought a new one which I intent to reread. When I first read it in 2003, I had this amazing feeling afterwards but could not quite attribute it to certain elements and topics in the book. The book as a whole has a lot of qualities, but this time I’m going to keep a little log on what resonates with me. I will try to write those down here for future reference.

Over at Marktplaats.nl we have developed one of The Netherlands’ biggest search engines (we’re doing so many searches per day, and making it easy for buyers to find the things they are looking for is considered “core business”). A post by Tim Bray actually summarizes nicely the basics of a search engine.

The post is already 4 years old (posted in late 2003) but is still quite relevant. It focusses primarily on filtering to a relevant resultset, although it doesn’t ignore sorting there is so much more one can do in that area nowadays.

For those that are absolutely new to this area, or find them selves playing with this type of technology but want to read up on it I especially recommend the following articles:

• Backgrounder
• The Users
• Basic Basics
• Stopwords
• Result Ranking (sorting)

And, before you think about breaking the market and build an intelligent search engine, please read this.

Excellent writeup, even 4 years later. Thanks Tim!

Another post, more related to relevance and the order your results will appear the relevance you can attach to users click behaviour. In short, users are generally inclined to click on items at the top more frequenctly, so basing your relevance metrics on this you need to “un bias” your data.

Secondly, as users get further down the result set -in the aggregate- they are going to switch to a different mode of selecting the items they are going to click on: they will actually start reading the excerpts and decide, based on the information present on the result set, on which items to click. In other words, here you should actually not try to “un bias” your data!

Kyle Wilson wrote recently a really nice piece on why software development is so hard, which for me didn’t include new insights (I’m already convinced) but did a really nice job on quantifying the problem space. Something which I had not seen before so clearly articulated. If you’re in this line of business, it’s a must read.

The thing that makes this article so interesting is that for some reason Kyle has access to information about five large software development projects: Chandler (the OSS Exchange replacement), Myst Online, Fracture (a new game), the software that controls a F-22 fighter jet and the FBI’s Virtual Case File.

After describing some of the pitfalls the Chandler team fell in, he goes on trying to outline why Lines of Code (LOC) is a useless metric for determining the complexity of a software program. More importantly, he throws in some statistics of the aforementioned projects that really hits this home.

Short list of conclusions:

• LOC is useless as a means to describe either the complexity of the program or the amount of effort that went into producing it
• Project teams need an economic framework (in the broadest sense of the word) in order to be successful. Otherwise there is no forcing function for decisions (like design choices, feature sets and release dates).
• In theory the complexity of a well-structured program should be O(n), where n is the number of lines of code (each line only tightly coupled with the line preceding and after it). A poorly structured program would be O(n2), with dependencies on one particular line throughout the codebase.

Favorite quote, from the 1968 NATO Software Engingeering Conference: “We undoubtedly produce software by backward techniques. […] We build systems like the Wright brothers build airplanes — build the whole thing, push it off the cliff, let it crash, and start over again”.

And this one: “Most software today is very much like an Egyptian pyramid with millions of bricks piled on top of each other, with no structural integrity, but just done by brute force and thousands of slaves” — Alan Kay (the father of Smalltalk).

Vogels (the CTO of Amazon) published a paper in which they describe their high available, eventually consistent data storage called Dynamo that will scale incrementally. It was an excellent read, and if you’re in the business of providing a high traffic, high available application (web or otherwise) I suggest you take a look!

That post did re-iterate with me a point I came across before: why is it that a company like Amazon is building these types of infrastructure components? There are other examples like it, providing excellent world class technology within eBay. Or more publicly why did LiveJournal.com develop memcached or Mogile? Why did Google write GFS? And the list goes on. This, by the way, is not just pertained to storage solutions. Within eBay I see some really cool technology that could be spinned off into separate products in different area’s, but I am not in a position to disclose those.

I do see that having such a technology could be a competitive advantage (for a while) but at this scale I’m not sure that that really holds. For example both Amazon and Google currently have a highly scalable data store (Dynamo vs GFS). (They are a bit different with Dynamo storing data smaller than 1MB)

Those technologies are really cool, and scratch an itch that is absolutely there for these companies but bottom line eBay, Google, LiveJournal should be adding features and improving the user experience above writing infrastructure components. Now, in order to either a) write those features or b) bring down operational cost (or availability up) you might need these technologies but that does not translate 1:1 into actually writing them. Ideally, an Application provider such as Amazon should be able to come up with a cool feature, buy the technology needed to back that feature up and develop the feature using the technology bought.

Now, why is it then that no 3rd party vendor stepped into this space and provided similar technology? Why is it that noone from these companies started off on their own and started a company providing a technology like Dynamo? Why doesn’t a big database or storage vendor step into this space? Clearly there are some big companies out there that need this technology (Amazon, eBay, Google, Yahoo, and there are certainly more). So, really, why has nobody stepped into this space? Or, in reverse, which companies provide these types of products?

I followed the iPhone introduction closely. Partly because I just love Steve Jobs to present, due to how he engages you (even if you’re watching the recordings) as well as the paradox of the casual Steve with overall slickness of the presentation and the products. But more importantly, I love Apple products. By now I own a black MacBook, 3 iPods and an Airport Extreme. (✓ Apple Fanboy… sorry)

Both the iPhone as well as the iPod Touch were greeted with a lot of enthausiasm and hype. In my mind, rightfully so, since they are great devices for their intent and above that superbly designed. However, as of late, I have seen more and more negative responses on the net primarily due to Apple restricting the platform both devices provide: Apple started to lock down the iPhone more and more to make sure you don’t unlock the phone and don’t install 3rd party software.

Now, I for one, would love an open iPhone/Touch which could be augmented by 3rd parties because either someone writes an app that proves to be usefull (more bang for the buck) or everyone screws it up and you end up not installing 3rd party software (seems unlikely, given enough time). This is not the point of this post though: Apple decided the platform is theirs, not yours. Applications written for both devices will come from Apple, and Apple alone.

This last bit is what pisses off a lot of technology folks. I’d like to provide a different perspective: buy the device if you’re okay with locked down platform. In the case you’re not: don’t buy it. Either buy something else (I wouldn’t know what though) or wait for Apple to see the declining sales trend. Because, believe me, if opening up the Touch/iPhone will increase sales substantially, Apple will do so.

But beware, there is another way this could play out: you’re waiting for Apple to open up the platform and sales are going through the roof. That just indicates that the mainstream audience doesn’t find the openness a necessity and that makes you a niche market. Given the current success (shopping season is around the block) you could be waiting a long time.

Bascially, given Apples reputation in terms of designing applications, operating systems and devices, they thought long and hard about the feature set, and if you’re not okay with that please change your perspective from “Why the ***** is this not an open platform” to “Jeez, I am apperently not in the target consumer group”.

Personally, I have a technology background myself, I am totally okay with the status quo: I’ve bought an iPod Touch the minute it came out and I’m loving it. It “just works” and that’s exactly what I bought it for. No need to tinker, no need to get some vague 3rd party app installed that will help me track my Todo’s, etc. I’m perfectly thrilled by the user experience as is!

There are however a few features that I either wish the Touch would have or otherwise hear some rationale about why they were not included:

• Wifi integration

The Touch is able to browse the web, buy songs on iTunes (I love exploring new types of music through the store, btw!), etc. But why doesn’t it see my MacBook on the network as well? It could do all kinds of nice features like:

Synch music/photos over the network without needing a lousy cable.
Synch my bookmarks: it’s quite irritating that I need to start from scratch in terms of bookmarks. (Although I find the keyboard doable after some practice, entering long URLs by hand is not my favorite past-time )

• Music/Photo

I can remove movies from the Touch on the Touch but I can’t remove any pictures or music. I find myself often using the Touch listening to music I really don’t want to have anymore, I’d like an option to remove those and after I synch it with the host computer they should optionally be removed there as well.
The titles of photo albums are quite large, which is great for readability, but there is no way to look at the longer album titles to figure out which one you’re looking for. For example, I have 6 albums called “Westendorf Wintersport 200x” and I need to determine by the photos which album I’m looking at… not great.
Synching smart play lists: I’ve got this “100 songs most played” play list which I turn on when I don’t want to bother with handpicking the song I listen to. The Touch does synch the playlist (I’ve got one on my Touch with the same name and filtering) but doesn’t load in the same numbers, causing all kinds of random numbers to enter into that play list which doesn’t seem necessary to me.

• Browser

Once, I tried to open up a few wikipedia articles in tabs to have some information handy during a recent holiday; only to find out that the Touch doesn’t keep it in memory and as soon as you try to open up that tab tries to reload the page. It would be great if it just kept a cache for those pages that are open in tabs (can’t be that hard, right?) so that you can still use them even if you don’t have wifi. Now that I think about it, it would be great to store YouTube video’s in the Movies section as well, given that you’ve already downloaded it.
One non-feature request: the safari browser intergration is superb; the best browser experience I’ve seen on a mobile device!

(Please note, that none of these issues with the possible exception of the Safari bookmark synch, would be solved by opening up the platform)

Lately a discussion on the memcached-mailing list has started where for example the guys behind facebook.com and bloglines.com are participating and sharing some of their experiences. I’m don’t think this is rocket science, but I’d like to quote some of the things that are being said and provide some links to the relevant discussions.

About the general “would you want to bet your uptime on memcached as an infrastructure component?”-question:

We consider memcached a critical part of our infrastructure. The benefit of memcached in a typical setup is to reduce the amount of database hardware you need to support an application; if you have enough database horsepower to run unimpaired with most of your memcached servers out of service, then there¹s probably no point using memcached at all, since it without a doubt adds extra complexity to your application code. [link]

If you shard all you data, etc. etc., is memcached still worth it?

Question:
And you would split (federate) your database into 100 chunks (the remaining 100 would be hot spares of the first 100 and could even be used to serve reads), wouldn’t that take care of all your database load needs and pretty much eliminate the need for memcache? Wouldn’t 50 such boxes be enough in reality?
Answer:
Don’t forget about latency. At Hi5 we cache entire user profiles that are composed of data from up to a dozen databases. Each page might need access to many profiles. Getting these from cache is about the only way you can achieve sub 500ms response times, even with the best DBs. [link]

Also, there is a lot of talk about a FUSE (File system in user space) filesystem based on top of memcached. Not only would that make caching available for those applications you do not control (blackbox) but it would have some really great advantages for your generic PHP app:

Over the last two weeks i spent a lot of time discussing a memcachefs (fuse-based) with two fellow geeks - applications that came to mind were (a) the smarty cache (b) php sessions; for both cases, losing files (as a whole, not random parts inside) is ok and readdir is irrelevant, which allows cutting a lot of corners. [link]

Terry Chay over at “The Woodwork” has a length but nicely written blog post about a PHP vs Ruby on Rails discussion. If you’re interested in that kind of stuff, read the article: it has some juicy humor sprinkled into it as well; it’s a bit flame bait too…

Favourite quote (quoting another quote):

“First they ignore you, then they laugh at you, then they fight you, then you win.”
—Mahatma Ghandi

OSCON 2005:

“Unless you’re Ruby.”
—Danny O’Brien, “On Evil”

And:

I can’t speak for Alex, but what I’m saying is look at the top 100 websites on the internet: about 40% of them are written in PHP and 0% of them are written in Rails. (Yes, I can (and am) using this statistic to grind you Ruby fuckers into the dust.)

Just watched the Lost season finale and I must say I absolutely loved it. So many variables, plots, and personalities changing around in the matter of an hours worth of television. I am one of the Lost-fans that actually liked pretty much all the episodes since the beginning — but even those that didn’t like the last season too much were blasted away by this seasons’ finale.

One interview that I’ve spotted online with Damon Lindelof is pretty interesting if you’d like a few answers to the many questions being posed by the finale.

Only 48 more episodes to go, since the show is slated to end in 2010!

Normally, I just follow my own Bloglines account, checking those blogs that I’m subscribed to. I always try to force myself skip on as many posts I can. But every once in a while I go on a rampage in search of new good blog (-posts). Here is a round up.

• Top ten things ten years of professional software development has taught me: lists the 10 things the author thinks software development has thaught him. My favorite: “he business likes to say that all the features are as crucial. They are not. Push back and make them commit.”
• Here is a post on Particletree (bookmark that site!) that talks through the 4 ways of delivering JSON objects from the server to the browser client. Also this other post that talks through paging through JSON/Ajax data plus preloading the data.
• A nice aggregation of articles talking about business models on the web
• Update: another really nice post on Amazon’s technology and what the advantage of using their systems give. (I’m not completely buying into it, but thats a post for another time — Microsoft shoving all their apps onto Amazon and Amazon not going to budge underneath the pressure??)