6diagrams.com is now socialinks.org

Considering how rapidly 6diagrams.com becoming less of a forum and more of social bookmarking, it needed a new name.

Domain buying experience is quite frustrating. All the good names are both taken and expensive. But after many tries, we finally found socialinks.org, which describes the application quite well.

With new year and new name, socialinks.org will be even more fun and enjoyable. =D

Reddit Web Server

As of late November 2009 – December 2009, It appears that Reddit is using Tornado web server. Interesting.

You can get the same type of information by using curl (curl -I http://www.reddit.com)

Source:

• http://toolbar.netcraft.com/site_report?url=http://www.reddit.com

Does Agile work?

This is question has been asked for many years now. And let me tell you right now, I won’t preach about it in this post.

Trying to understand Agile has been very difficult for me. Agile defines many things,  it is a collection of techniques and methodologies, it is a subset of software industry, and it is an overly positive adjective. It’s been quite a journey between me and Agile, and this post is my way of sharing the experience and as you will find out later, I’m struggling in some of them.

Short Terminology:

• Customer: Can be your clients or management.

The following are the ones that I’ve been exposed to, definitely not a complete Agile list.

(Technique) Pair programming

There are enough people who said that pair programming may not be effective, and not for everyone. The reason is that, while 1 programmer is coding, the other one is bored to death.

It’s easy for me to see why other people can have terrible experience while pair programming. Even though pair programming sounds easy, there are a lot of hidden requirement behind pairing:

• Both programmers need to be at about the same speed/skills, kind of like badminton.
• If one is a much better programmer, then s/he needs to have a good verbal communication skills (another implicit requirement). There’s no way for slower programmer to keep up with vim windows without verbal guidance.

I see that pairing is valuable because it distributes risks. Some programming decision making is far riskier than others. By having another programmer to weigh in other options, I feel less stress already.

I have no doubt in this technique. Should I do it all the time? I don’t think so, many tasks are simple enough to be done solo.

(Technique) Write tests

A few popular programmers said that writing tests slow them down. I could see that, because is a different way of doing things than they are used to. And since startup software development cycle is quite harsh, picking up a habit of writing tests will always be low in priority.

To me, writing unit tests have 1 obvious value, to prevent regression. I am willing to sacrifice speed in favor of stable code base. Nobody like to fix the same damn bug introduced by different (in few unfortunate case, same) people.

How about other tests such as integration or behavior-driven? This is the part that trips me. When it comes to integration tests, how integrated should it be? All the way up to javascript layer? Launching browser to run tests isn’t exactly a speedy experience.

I also struggle with behavior driven tests. Should I always NOT test implementation detail? Many times, I’m interested in how the implementation actually works.

(Technique) Write tests first

I struggle with this hands down. Writing tests first is like asking me to write with left hand and I’m a righty. I need to understand the problem domain and perform a couple of spikes to further understand it. And I cannot do that if I have to write tests first. Because of this I can never call myself a TDD programmer.

(Process) Quick feedback cycle

From Agile perspective, customer should have quick feedback cycle so that they could be flexible in their planning. I agree with that. It is unrealistic to set a far reaching, fixed schedule in software world because there are too many moving parts (new features or solved problems in the past).

That said, I’m looking at this process with a grain of salt. What’s stopping the customer running around in circles accomplishing menial successes? It’s of course the developers job to remind the customer but alas, often times we are too busy duct taping.

(Process) Planning game

I like this process. This is the most concrete, easiest way to inform customer whether xyz is feasible or not. There are many tangible benefits that come from planning game:

• Customer is aware of development progress.
• It’s easier to ground customer to reality because they are aware of what developers can or cannot do given velocity X.
• It invites customer into creative thinking process because they need to think hard in choosing their features.

(Process) Estimates velocity

IMHO, any developers who are doing contract work need this skills. From business perspective, that’s pretty much the whole point. How can developers profit from selling services if s/he has no idea how much s/he can produce in X time?

This skill is also important for full time developer to set expectation to management. By knowing your velocity, you can prevent your own burnt-out.

Implicitly, this skill gives developer ability to say no to overly ambitious schedule. Does customer always listen to your estimates? probably not. If they never agree to your sensible estimates, then your alarm bell should ring.

(Process) Collaboration

Working in team setting is heavily underscored in Agile. Pair programming is subset of this. In general, this makes sense. Cooperation of many people almost always yield greater results. But collaboration is not silver bullet, there’s implicit requirement to have a functioning collaboration.

Most importantly is trust. It’s impossible to have any kind of collaboration if there’s no trust among members of the team. Even in family setting it wouldn’t work. For companies that have distrust or paranoia environment, they have bigger problem to solve and I don’t think Agile could help.

(Meta) The simplest thing that could work

I believe this is why we are paid to do programming, to exercise our technical judgment in creating simple solutions. Any type of users/customers always demand simple solutions even if the requirement is complex, Mint.com is a great example. I believe this mindset is useful even outside Agile context.

(Meta) YAGNI “You Ain’t Gonna Need It”

YAGNI is the subset of “the simplest thing that could work”. YAGNI mindset can help developers to avoid becoming architecture astronaut. It’s far too easy to architect something that’s far in the future, especially if the development team is shielded from financial reality. It’s dangerous for development team to spend too much time designing for issues that may never arrive (this issue might be more relevant to startups). Again, I think this mindset is useful in general.

6diagrams.com: Create your own forum.

I’ve been building this little thing for a while now. That little thing is Forum software.

It does everything you would expect from using PHPBB, Reddit or HackerNews, and more. With 6diagrams you can bookmark that interesting topic without having to go through your piles of links in that bookmark tab. It also, of course, remember your own posting.

To learn more of what it can do, please follow this link.

Why?

Already 3 guys asked me this questions. Why are you building this thing that looks like Reddit or HackerNews rip off? Well… in my defense, Reddit, HackerNews, (and Google) do a great job in displaying list of interesting things effectively. It’s not a coincidence that I’m following the path of Reddit/HackerNews because I want to follow the path of success.

Besides that UX decision, I build this because I’m starting to have difficulties when blogging. The standards of blogging is going higher and higher. Blogging takes more than just well-written article these days. I have to stay consistent in following the blogs general theme. If I want to write slightly controversial topic, I better do a lot of research and write it in concise manner. When my post is mildly interesting, I have to be vigilant in responding to visitors in less than 24 hours time frame. It’s tough (I believe this is 1 of many keys to Twitter success, it’s very personal).

Those are too much burden for a programmer who have probably only 1 hour a week to write a blog post. Sometimes I just want to voice out my opinion regardless of the accuracy (I never policed any of my real life friends when it comes to factual accuracy, I’m hoping to experience the same thing online). I want to write something that’s interesting enough to create a conversation for social interaction and personal knowledge.

That’s why I defaulted back to Forum format. Forum is great, it doesn’t force people to use real name. Participation in a thread can extends to weeks, so that I have time to respond to conversation (just like mailing list).

Most forums have informal, laid-back conversation about common things that people like, such as OSS project, anime, coffee grinders, Mitsubishi Evolution, etc. There’s no pressure to be right, everyone is simply sharing and hanging out.

That’s why I build 6diagrams.

Technical

6diagrams is open source forum software. The project page can be found here. It is built using Python, Pylons, MySQL, TinyMCE, and Tokyo Tyrant.

For those who stopped by and created accounts, I hope you find 6diagrams enjoyable.

Pylons: production environments

First of all, there are several wiki posts at pylonshq.com already, below are some of them:

• http://pylonsbook.com/en/1.0/deployment.html
• http://wiki.pylonshq.com/display/pylonscookbook/Running+Pylons+with+NGINX
• http://wiki.pylonshq.com/display/pylonscookbook/Running+Pylons+with+SCGI+and+Lighttpd

I believe there’s not a single best solution in choosing any of these strategies. This post is intended as sharing my experience in some of these. I’m definitely not set on particular one and will change my mind once I gained better understanding.

My testing methodology

I’m currently testing multiple deployment configurations on the same app, thus giving me the opportunity to blog about it. I use Apache Benchmark (on client side), top and ps afx on server side. My machine is 1 Linode 1440 instance.

My AB setup are:

ab -n 500 -c 50 -k http://rootapp.com/

ab -n 1200 -c 50 -k http://rootapp.com/

ab -n 1500 -c 50 -k http://rootapp.com/

ab -n 800 -c 800 -k http://rootapp.com/

The results varied insignificantly with some subtle interesting differences. I’ll explain those below.

Note: I am testing these configuration on dynamic AJAX-y web application, thus reporting hard numbers is not very useful.

CherryPy vs Paste HTTP Server behind NGINX

One thing  that I noticed immediately is that CherryPy has better performance than Paste’s HTTP server. On both, having multiple processes does not help much on overall performance, but significantly reduces number of failed requests. When run under multiple processes, CherryPy consistently have the least number of failed requests.

For my setup having 7-8 processes is the sweet spot. When I have more than that, top is telling me that the latter processes are under utilized.

Setting up CherryPy on your production.ini is painless:

use = egg:PasteScript#cherrypy
numthreads = 20
request_queue_size = 512
host = %(http_host)s
port = %(http_port)s

By just comparing the two, CherryPy is easily the winner.

Lighttpd and SCGI

This gist is basic configuration to get SCGI up and running on lighttpd while the following is setup for your production.ini:

use = egg:Flup#scgi_thread
host = %(http_host)s
port = %(http_port)s

given the same AB configuration as CherryPy and Paste counterpart, lighttpd and SCGI consistently capable of handling 30 requests/seconds. About 8-10 requests/seconds more than CherryPy. Even though this setup is better, I noticed that memory consumption continues to go up after 2 weeks. I haven’t spend much time in investigating why. The reason I didn’t choose this path is more because I simply like NGINX better.

If only SCGI module on NGINX isn’t so experimental.

NGINX and FastCGI

This gist is basic configuration to get FastCGI running on NGINX while the following is setup for your production.ini:

use = egg:Flup#fcgi_thread
host = %(http_host)s
port = %(http_port)s

With this configuration, I consistently get about 25 requests/seconds. It’s a bit behind lighttpd and SCGI configuration. Interestingly, when run under ab -n 1500 -c 50 -k, this configuration hangs NGINX requiring it to be restarted. It only happen once though.

Again, when load balanced properly (depending on your app), any one of these configurations would work well. Hopefully this post can help others to get up to speed in Pylons deployment.

HTML Default Colors

Someone dear to me put together this not too long ago:

pink, lightpink, palevioletred, hotpink, deeppink, red, tomato, crimson, firebrick, indianred

darkred, maroon, brown, sienna, saddlebrown, rosybrown, tan, darkkhaki, BurlyWood, chocolate, peru, darkgoldenrod

lightcoral, coral, lightsalmon, salmon, darksalmon,
orangered, darkorange, orange, sandybrown, goldenrod, khaki, gold, yellow

greenyellow, lightgreen, lawngreen, chartreuse, lime, springgreen, mediumspringgreen,
limegreen, green, forestgreen, darkgreen,
seagreen, mediumseagreen, darkseagreen,
olive, olivedrab, darkolivegreen

azure, aliceblue, lightcyan, paleturquoise, lightblue, lightsteelblue, powderblue, cyan, aqua, aquamarine, turquoise, lightskyblue, skyblue, mediumaquamarine,
mediumturquoise, darkturquoise, deepskyblue, cadetblue, cornflowerblue, steelblue, slateblue, mediumslateblue,royalblue, dodgerblue, lightseagreen, teal,darkcyan, blue, mediumblue,darkslateblue, navy, darkblue, midnightblue

indigo, blueviolet, mediumpurple, mediumorchid, purple, darkmagenta, darkviolet, darkorchid, lavender, thistle, plum, violet, orchid, magenta, fuchsia, mediumvioletred

darkslategray, dimgray, gray, slategray, darkgray, silver

Pylons: Tokyo and Redis for caching backend

First of all, I love Tokyo and already use Tokyo as secondary database for Pylons development and so far it has been a great success.

Since my box does not have a lot of memory, and a lot more disk space, it make sense to use Tokyo as caching solution instead of memcache.

Quick googling revealed that Jack Hsu has already implemented beaker’s Tokyo extension. His snippet works out of the box.

For my use case, I change the serializing strategy to using pickle instead of json. My reasoning is that pickle allows serializing complex object and I don’t have requirement for portability on cache data.

You can find the Tokyo extension here. I added Redis extension as well since it is very similar to Tokyo.

Edit (08/26/2009): Added extension for Dynomite

Edit (08/26/2009): Added extension for Ringo

[fixing bad layout]