Pylons Cheat Sheet

http://workaround.org/pylons/pylons-cheatsheet.html

SqlAlchemy in Three Minutes

The documentations of SqlAlchemy gets better in every version, but I can never remember the steps of creating Engine, Index, and Session.

So, I created 3 minutes tutorial on how to define the ORM class using declaration_base(), build that table, populates, and fetch.

See code here.

Python: Object to JSON

I want to store python object to JSON. Including a complex one (not just object that contains simple attributes).

Apparently, John Paulett has a need for that too. Here’s his code. The best part is: It uses CJSON! Bingo.

Python: using Memcache Client

For some reason, it’s always hard to find example on how to create memcache client object using Python.

This time, I will remember:

import memcache
memc = memcache.Client(['127.0.0.1:11211'])


Ruby: Moneta

In short, Moneta is just like Shove in Python. It is a single interface to various dictionary/hash like storages. Which means all those sexy Key-Value databases.

I’m currently benchmarking Moneta’s Tokyo vs Redis vs Memcache.

Results so far:

• Surprisingly, the Tokyo implementation is significantly faster. Even against memcache implementation. Why?
• Similar to LightCloud benchmark, the size of value does not affect speed of storing/getting.

Reference:

• Source Codes

Python: Drinking the Tokyo Kool-Aid

After reading what LightCloud can do, of course, it’s only natural to create object that serialized to Tokyo.

And that exactly what I did. The project (called Hail) is still infant, but the profile tests already answers some of my questions and curiosity about LightCloud (and Tokyo).

One obvious weakness I need to tackle: Serializing is too slow.

Questions that got answered:

• Slowness is not caused by the size of the object, instead it is caused by number of items.
• LightCloud does execute a lot of function calls. Most of the are really fast though.
• EDIT: Tokyo is fast! Especially after I compare it with Memcache. But LightCloud is not. Tokyo is not as fast as I thought… but this is not final thought, I should create profile_test on raw tokyo tyrant node. On top of that LightCloud overhead is not negligible.
• Serializing to cjson is faster than cPickle. That’s surprising.

Next, I should test getting items from both memcache and tokyo. I’m expecting it to be really fast.

References:

• PyTyrant benchmark
• Tokyo Tyrant benchmark using Ruby + Rufus + Moneta
• Lightcloud benchmark
• Lightcloud site
• Hail GitHub
• More about Tokyo Cabinet

Beards of Python

I believe these are PyCon09 attendants.

How fast is Python dictionary?

Based on textbook theory: O(1).

I was about to do a profile test on it, but found this discussion on the mailing list. One poster claim 0.2 seconds per 1 million keys.

Perfect.

Even Matz love Python

Matz love Python

Python: cPickle vs ConfigParser vs Shelve Performance

I need to store large number of key-values to map my Python objects. These key-values DO NOT have to be replicated across multiple servers and  the project DOES NOT require external storage systems such as RDBMS or Berkeley DB or others. The least external dependencies the better.

That leads me to cPickle vs ConfigParser vs Shelve. cPickle is obvious contender, it is fast and easy to use.

ConfigParser is an interface for writing config file, but its format is very key-value ish, so it counts.

Shelve is obvious too, because of its interface.

So I ran profiler test using hot shot and here’s the result:


Profile: Saving 100000 key-value to pickle file
700001 function calls in 2.330 CPU seconds

Profile: Extracting 100000 key-value from pickle file
4 function calls in 0.258 CPU seconds


Profile: Saving 100000 key-value in ConfigParser file
900004 function calls in 2.502 CPU seconds

Profile: Extracting 100000 key-value from ConfigParser file
300007 function calls in 1.936 CPU seconds


Profile: Saving 100000 key-value to shelve file
1300047 function calls (1300045 primitive calls) in 10.091 CPU seconds

Profile: Extracting 100000 key-value from shelve file
500027 function calls in 6.527 CPU seconds

From the results:

• Shelve is disappointingly slow. It execute 1,300,047 calls???
• cPickle is not bad at all. As expected, it performs really quick.
• ConfigParser is the biggest surprise here, I was expecting it to be much slower.

Side Notes:

• I use threading.Lock before setting the key-value to prevent resource contention (which is real life case).
• Any improvements is greatly appreciated. Especially different data storage that I’m not aware of.
• Code can be found here.