Introduction

As discussed in a previous thread, http://support.zenwalk.org/viewtopic.php?f=3&t=22066 I had decided to become a person without a history, at least with respect to Iceweasel. The whole question of what went on in the places.sqlite database had me interested though. If I had got rid of all my history and I had VACUUMed the residue out, why was I still left with with a 600kb places.sqlite file. Thats rather a lot of nothing, even for a database, so I went poking round inside it.

SQLite Manager

Fortunately, given my sql skills, there is an Iceweasel addon, SQLite Manager https://addons.mozilla.org/en-US/firefox/addon/5817 that allows you go into the databases behind your Iceweasel browser and have a poke around. Again, its important to stress, that while you are learning to use this tool, you work on a COPY of the relevant sqlite file, so that you don't muck up anything vital.When you connect to the database, just give the application the absolute pathname to the COPY of the sqlite database file you want to explore, and it will open all the tables for you.

In this case, the database file of interest is places.sqlite. A diagram showing the interelationship between the various tables in this database can be found at http://www.forensicswiki.org/index.php?title=Mozilla_Firefox_3_History_File_Format and the file we are interested in at the moment is the moz_places table. This table holds the references to each of the sites you have visited and that are currently held in your History File. Further details relating to your history are to be found in the moz_historyvists. These two tables share a common value to tie the two tables together ...... moz_places.id = moz_historyvisits.place_id ..... and this can be seen in the diagram by the line linking the two values together.

My Own Bookmarks

Now since I have zeroed my history, what else is there of interest in this table that would help explain the 600k of data. Well there are all my bookmarks hiding in here too Further information on my bookmarks is held in the moz_bookmarks table and the link between the two tables is moz_places.id = moz_bookmarks.fk. What this means in practice is, if you find a bookmark of interest in the moz_places table, take a note of its id number in the first column of the table, then go into the moz_bookmarks table, sort the table on the fx column, by clicking on the title of the column, then look for the same number in that column. When you find it you will have the additional data relating to that particular bookmark.

Another column of interest in the moz_bookmarks table is the parent column. If you sort on the parent column you will notice that there are groups of bookmarks with the same parent number. This parent number corresponds to the folder in your Bookmarks that you have put these various bookmarked references into. Now if you go prowling a bit further you will notice that the bookmarks that you made with the old method of bookmarking, and that are currently unsorted, each have a unique parent id number in a particular range. Also that your unsorted bookmarks that you have made by clicking on the star in the Awesome Bar, will each also have an unique parent id number but in a much higher range. The third column of interest is the moz_bookmarks.position column that shows the position of a bookmark within a particular folder (parent). So an individual bookmark can be referenced by three vaues moz_bookmarks.fk, moz_bookmarks.parent and moz_bookmarks.position. Now in terms of database performance the search for a particular bookmark is going to be improved if you dont have large numbers of unsorted bookmarks of either style floating around in the database. So a few minutes of using the Iceweasel Library function in Bookmarks (Organize Bookmarks) to drop your unsorted bookmarks into folders will improve how your database works in terms of performance and its also easier for the user to find a particular bookmark as well.

Live Bookmarks and Performance Hit

So having accounted for the references to my own bookmarks in the moz_places and moz_bookmarks folders, what else is there in there of interest. Well it turns out that in the moz_places table there is a reference to each of the Live Bookmarks (RSS Feeds) that I have added to Iceweasel and in the moz_bookmarks table there is a reference to each of the articles you can currently see in each of your feeds, grouped together under a single parent id number for each RSS feed.

Now this is where your performance hit actually happens. When Iceweasel starts up, each of the RSS Feeds has to be synchronied and the new data written into the moz_bookmarks table after deleting the old references relating to a particular parent (RSS Feed). Now this is a relatively CPU and memory intensive operation that has to be done for everyone of your Live Bookmarks everytime you open Iceweasel. Now I had 30+ Live Bookmarks, most of which I never looked at on a daily basis, and could have upto 40 odd references in some instances. So each time I started up Iceweasel I had to go through all of the database deletions and additions before I was fully operatinal. .On my reasonably tuned but aging system it was taking close to two minutes for Iceweasel to be fully operational. Sluggish wasn't the word!! I also found out that Iceweasel does this all over again every 60 minutes (default) as it refreshes all the Live Bookmarks.

It turns out this is a known problem and the following links may be instructive

http://getsatisfaction.com/foxmarks/topics/slow_performance_with_foxmarks_and_firefox_3_0_5

https://bugzilla.mozilla.org/show_bug.cgi?id=329534

http://www.mydigitallife.info/2008/07/13/change-the-default-reloading-time-of-firefox-3-live-bookmarks-feature/

Possible Solution

Now to be honest, I'm more interested in a fast and responsive Iceweasel, than I am in waiting around for it to load up a whole load of Live Bookmark s(Rss Feeds) that I may not read from one week to the next. So for me the answer was obvious, find someway of reading my RSS Feeds that wasn't going to be a drain on my system resources and once I had done that, delete them from Iceweasel and improve my performance of my browser.

Now before I delete them all, I do want to have a copy of their relevant URLs so I can reconfigure some other application later. If you click on the relevant RSS Feed in the Organize Bookmarks Library window, it will give you the relevant URL for the RSS feed, which you can then copy and paste into mousepad for later use. Once you have all your references, right click on each Live Bookmark and delete each one, then close down Iceweasel. Now run the vacuum over the sqlite database files, as discussed in the previous post, to clear up any free space in the database tables, and then restart iceweasel.

Results

There were two main results for me out of this exercise. Firstly, I now have a places.sqlite database file that is only 300Kb in size, which contains only the data I want to be in there, and which loads reasonably quickly and isn't re-written every 60 minutes. Secondly, my Iceweasel now starts in about 25 seconds from a cold start, which ain't bad for a PIII, given the number of addons it also has to load. Its overall response time is also much faster, which may be due to my doing some maintenance on the underlying database eg sorting out my bookmarks into folders and vacuuming the database. Iceweasel now loads in about 60 MB of RAM instead of the 95 MB of RAM previously, which on a system with only 378 MB of RAM overall is a great saving.

And my RSS Feeds (no-longer Live Bookmarks) well they are now running in Canto, which is a terminal based RSS reader which we have been discussing in #zenwalk. The home page is at

http://codezen.org/canto/

Canto gives me all the functionality that I want, is far more confiurable than anything I had on Iceweasel, so I have a new toy to fiddle with, uses minimal system resources and it opens anything I want to read in a tab in Iceweasel with a single keystoke.

The following link may be of some use. It explains how to code a query into the Iceweasel Database.and to use the information within an application. I like the idea of the application creating a temporary copy of the relevant database file to work on, thus avoiding the Iceweasel database lock problem, and then deleting the file when nolonger required by the application. In my situation, I would work with the moz_bookmarks table to obtain the required bookmark.

http://www.tuxradar.com/content/hudzilla-coding-academy-project-seven

Hope this all proves to be of use to people

Regards

Michael