In case this is useful for you, I am currently finishing a new version of my Python parser. I tested 7zip decompression several types, piped to the parser then piped to MySQL. I can confirm that 7zip paralellizes decompression much better (I use 2 AMD Santa Rosa 2 GHz, 4 cores in total, 3Ware RAID-6 with 8 Seagate fast SATA-II disks).

Big Wikipedia dumps crash (no matter the language edition) around the 4.500.000 revision mark when piped, and exception control is required (so I now forget about pipes, and use a library for directly conecting the DB). Therefore, chunks will be more than welcomed. Incremental back-ups will be far more flexible than the current solution.

I already take advantage of extended inserts in the parser, to speed up even more the recovery process.

- wget part1
- analyze part1
- wget part2
- analyze part2
- …
- merge data

And sorry for waiting for respond.

From what Brion told me in another discussion, much of the time required for a dump is actually taken up by the compression phase. The actual dump takes only a couple of days.
Since the Feb 6 07 dump already has been compressed, I think it would not be necessary to produce a single compressed dump after that — monthly compressed incremental dumps are all that is required.

You already built a mechanism to append new data to an existing dump. I assume that saves a lot of time, as it saves tremendously on SQL calls. All those joins should mean quite a few I/O’s per revision dumped, right? of course mitigated by cached indexes. Yet I wonder:

Q1: Is this incremental dump system used often or does it have practical obstacles?

Q2: As reading a full dump takes about one day only, a normal full dump (now 6 weeks for the English Wikipedia) is mostly SQL calls right?

————–

I could see at least two ways to split up the archive dump and make the process more manageable, at the cost of being somewhat more complicated:

1 Produce chunks that contain exactly one calendar month of data, which is preserved as a separate entity ever after. Every new dump needs to collect data not collected before and complete the latest unfinished month and start a new one.

A separate queue of article and revision deletions (keys only) is updated on an ongoing basis (appending keys for new deletions). So for every month there is a never changing dump, and a growing delete queue, to be merged on a researchers PC.

This way every month needs to be downloaded only once, and for each month a thousands times smaller deletion queue needs to to be redownloaded and reapplied occasionally (of course in an automated fashion).

One downside might be that users do not apply the latest deletions. To this I would say it is unavoidable anyway that users keep old dumps that contain officially banned records, if they wish to do so. The well behaved way is mainly there to get proper data for research, stats and fork purposes.

I know the total size of the dumps would grow due to less optimal compression, as bad side effect. But this system greatly shortens both dump and download times.

2 Another way to make the process better restartable would be to produce a new dump every time but write to a new file for every 100,000 articles. The current English dump would result in +/- 25 files. This would not save on dump time, but greatly simplify disaster recovery/restartability. The process might even be distributed over several servers, each doing a part of the English wikipedia.

In both scenarios it means either that
- existing scripts (like wikistats) need to be adapted to handle multiple input files or
- a new script needs to be built to merge chunks to an archive file as we know it now.

—-

unrelated: I sent you a mail several weeks ago on two addresses. Did you get any? If not please mail me and I will respond to that. Thanks.

That’s not a rhetorical question, I really want to know so we can make this process better for the people using the dumps.