Fun - lootmar software

This one I have been liking a lot, always a fun for those Hindi movie lovers, the famous Sholey movie of Amitabh Bachchan.

Here the software version goes -

Gabbar sends Kaalia and two others to Ramgad to collect the loot-maar software he had ordered.

They reach Ramgad and started shouting: "Abe O thakur! Kahan hai wo loot-maar software? Last date to kab ka nikal gaya ".

Thakur [with anger]: "Chillao mat! jaakar Gabbar se kah do ki Thakur Software walon ne paagal kutton ke liye software banana bund kar diya
hai."

Kaalia: "Bahoot garmi dikha rahe ho thakur? Koi naye programmers hire
kiye hain kya?"

Thakur: "Nazar uttha ke dekh, Kaalia, tere sar par powerbuilder chal raha hai."

Kaalia looks up and sees Viru (Dharmendra) working on a PC on one
Water tank and Jay (Amitabh) on another, using a laptop.

Kaalia Starts Laughing and says: "Ha ha... thakur ne freshers ko liya
hai, Ye log Programming karenge? In ko to DOS commands bhi nahin
aate."

Veeru shouts: "Chup-chaap chala ja kutte. Hum log consultants hain,
Kuch bhi kar sakte hain."

Jay hits his keyboard,then says: "jaao kaalia, Gabbar se kahna ki uska
server down ho gaya ..."


AT GABBAR'S DEN...


Gabbar: "Kitne bugs the?"

Kaalia: "Do sarkaar."

Gabbar: "Wo do! Aur tum teen. Phir bhi fix nahi kar sake? Kya soch key
aaye ho? Gabbar bahoot khush hoga? Naya assignment dega ....aur
increment bhi? Iski saza milegi... barobar milegi."

[Snatches an X terminal from Sambaa]. "Kitne sessions hain is machine mein?"

Sambaa: "Chhey sarkaar."

Gabbar: "Session chhey aur programmer teen. Bahoot naainsaafi hai .."

[logout - logout - logout].

"Haan ab theek hai... ab tera kya hoga Kaalia?"

Kaalia: "Sarkaar, maine aapka code likha tha."

Gabbar: "To ab documentation kar!

Ha...... Ha...... Ha...... Ha...... Ha...... Ha......

Postgres- achieving high insertion rate in multithreaded environment

Suppose you have an application which runs multiple threads, create some set of data in memory, then all of them dump in one single table. This table is short lived. There are some queries etc done on it and then it can be dumped.

During our experiments, it was found that this was getting slow. What I did then, I created two installations of my applications, and measured the rate, and found that the insertion rate got increased. With the help of a DB expert, it came out that this happens because of transaction logs locks.

For example, your application has just one thread, it creates one set of data and then puts that in DB. With this case, you are able to achieve 1MB/sec insertion rate. Now if you increase the number of threads to 5, each of them will create one set of data, hence, total 5 sets of data, you would expect that you should be achieving 5MB/sec as threads are supposed to run in parallel. In fact, that does not happen as the threads now are blocked in locks on transaction logs. (we are assuming one DB connection per thread). Hence, the performance goes down a lot.

How to solve this?

In PostGreSQL 8.3 onwards, they have provided a mechanism for the same.

In above case, all the 5 threads need to write their data to some file, or one structure. Once the data is aggregated, since the table can be dumped, we will truncate it. So, begin a transaction, truncate the table, insert the data in table, and commit the transaction. In this case, since only on thread is going to write to DB, transaction logs are not enabled, hence no locking, and hence, we see a real good performance. The rate can be actually 5 MB/sec or more.

You would ask this - I can simply aggregate the data in memory and dump using one single connection, why to do all this? You are right. We needed to do this as we were using multiprocessing in python (as multithreading is not true multi-threading there), and the data could not be shared across processes, so, we had to write to some file on the disk, and then in one single transaction load that file in DB. We got a very high performance in terms of insertion rate.

Best luck.

Blogging??!!

Just setup my first blog, I am feeling excited about it.

I promise, it would be one of the blogs you would want to visit during your leisure time, or when you feel depressed, bored, being let down or so, and I promise it would make you happy and bubbling once you visit this blog.

It would hold all the nine 'rasa' (flavours) of life, I promise, but it would not hurt anyone.

I promise, it would help you hold your spirit high.