marcusmeissner ([info]marcusmeissner) wrote,
@ 2007-09-25 17:03:00
Previous Entry  Add to memories!  Tell a Friend!  Next Entry
Current location:Home
Current mood: amused
Current music:Random Playlist

speeding up zypp
On my Laptop, a 800Mhz P3 Mobile, 256MB RAM, ext3, the new updater has always been a bit slow.

After several beta upgrades I had it at RC1 level and tested online updates.

$ time zypper lu
real    3m4.187s
user    0m31.306s
sys     0m3.440s

Boringly long and it hits the disk constantly. Lets see the size of the zypp.db sqlite cache ... 80 MB (wtf? for the whole 10.3 repo + 2 times the update repo?)
Also the rpm database is 40MB for Packages, and another 10MB for the filelist (why?).

So what can we do:

Defragment on the disk


Due to repeated increase the file is fragmented on the disk. Not good for seek patterns.

The easiest way to fix this is to:
cp file file.new ; rm file ; mv file.new file
The full file copy defragments the file nearly by itself.

So doing this for the large zypp.db, Packages and the Filenames and Dirnames of /var/lib/rpm we
get:
$ time zypper lu
real    2m40.794s
user    0m31.242s
sys     0m3.400s

24 seconds saved.

Clean up the zypp.db cache


There is internal fragmentation in the zypp.db, partially caused by repos I had added and now
removed, partially due to refreshes.

So we run:
sqlite3 /var/cache/zypp/zypp.db vacuum
It is now 29 MB.
Alternatively you can also just rebuild it from scratch, which will give the same result
rm /var/cache/zypp/zypp.db ; zypper refresh

fragmentation of RPM Packages DB



The RPM DB also seems to get internally fragmented, the "reading packages" takes very long.

I tried rpm --rebuilddb.

Now the time is:
$ time zypper lu
real    2m2.523s
user    0m30.582s
sys     0m3.400s

And defragment it again (cp Packages Packages.new; rm Packages; mv Packages.new Packages)
$ time zypper lu
real    1m7.278s
user    0m32.730s
sys     0m3.160s


Just 2 minutes of 3 saved, and just a 200% speed up.

And SDBized: http://en.opensuse.org/SDB:Speed_up_Package_Manager_Stack

(Post a new comment)

W-O-W!
(Anonymous)
2007-09-25 07:33 pm UTC (link)
Really. Simply vacuuming the zypp.db and rebuilding the rpm db has sped up the second run of zypper lu from 46s to 36s where in both cases 20s is user (CPU-bound) time.
-mvidner

(Reply to this)
Great!
(Anonymous)
2007-09-25 09:41 pm UTC (link)
One might ask why we do not add something like this to zypp itself? The mentioned steps should be automated.

Timo

(Reply to this)

(Anonymous)
2007-09-26 08:51 am UTC (link)
very cool, thanks

(Reply to this)
Poor test method
(Anonymous)
2007-09-27 06:50 pm UTC (link)
I found that simply running the 'zypper lu' process twice in a row reduced the time to complete by 25% in my case going down from 0m48sec to 0m36sec. On a different machine running twice in a row without any of the defrag or db compression/rebuild steps improved the time from 1m18s to 0m36s.

So I think that this procedure of 'db compress/file defrag' needs more validation. So I propose:
Stage A. Run zypper lu 5 times, report the high, low and average times.
Stage B. Perform the 'db compress/file defrag'; run zypper lu 5 times and report high, low and average times.

Compare Stages A and B. I suspect that the speed up shown in the article may be due to reading files into RAM cache....

Cheers,

Ron

(Reply to this)(Thread)
Re: Poor test method
[info]marcusmeissner
2007-09-29 03:18 pm UTC (link)
Sure.

I did run zypper lu before the first test call.

(Reply to this)(Parent)

Create an Account
Forgot your login?
Login w/ OpenID
English • Español • Deutsch • Русский…