sharpetronics/PostgreSQL
1
0
Fork 0
mirror of https://github.com/postgres/postgres.git synced 2025-11-04 00:02:52 -05:00
Code Issues Projects Releases Wiki Activity
PostgreSQL/doc/TODO.detail/pglog
Bruce Momjian d07766fb50 Update TODO list.
1999-10-18 01:12:51 +00:00

2901 lines
133 KiB
Plaintext
Raw Blame History

From aoki@postgres.Berkeley.EDU Sun Jun 22 19:31:06 1997
Received: from renoir.op.net (root@renoir.op.net [206.84.208.4])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id TAA19488
for <maillist@candle.pha.pa.us>; Sun, 22 Jun 1997 19:31:03 -0400 (EDT)
Received: from faerie.CS.Berkeley.EDU (faerie.CS.Berkeley.EDU [128.32.37.53]) by renoir.op.net ($ Revision: 1.12 $) with SMTP id TAA18795 for <maillist@candle.pha.pa.us>; Sun, 22 Jun 1997 19:18:06 -0400 (EDT)
Received: from localhost.Berkeley.EDU (localhost.Berkeley.EDU [127.0.0.1]) by faerie.CS.Berkeley.EDU (8.6.10/8.6.3) with SMTP id QAA07816 for maillist@candle.pha.pa.us; Sun, 22 Jun 1997 16:16:44 -0700
Message-Id: <199706222316.QAA07816@faerie.CS.Berkeley.EDU>
X-Authentication-Warning: faerie.CS.Berkeley.EDU: Host localhost.Berkeley.EDU didn't use HELO protocol
From: aoki@CS.Berkeley.EDU (Paul M. Aoki)
To: Bruce Momjian <maillist@candle.pha.pa.us>
Subject: Re: PostgreSQL psort() function performance
Reply-To: aoki@CS.Berkeley.EDU (Paul M. Aoki)
In-reply-to: Your message of Sun, 22 Jun 1997 09:45:31 -0400 (EDT)
<199706221345.JAA11476@candle.pha.pa.us>
Date: Sun, 22 Jun 97 16:16:43 -0700
Sender: aoki@postgres.Berkeley.EDU
X-Mts: smtp
Status: OR
the mariposa distribution (http://mariposa.cs.berkeley.edu/) contains
some hacks to nodeSort.c and psort.c that
- make psort read directly from the executor node below it
(instead of an input relation)
- makes the Sort node read directly from the last set of psort runs
(instead of an output relation)
speeds things up quite a bit. kind of ruins psort for other purposes,
though (which is why nbtsort.c exists).
i'd merge these in first and see how far that gets you.
--
Paul M. Aoki | University of California at Berkeley
aoki@CS.Berkeley.EDU | Dept. of EECS, Computer Science Division #1776
| Berkeley, CA 94720-1776
From owner-pgsql-hackers@hub.org Mon Nov 3 09:31:04 1997
Received: from renoir.op.net (root@renoir.op.net [206.84.208.4])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id JAA01676
for <maillist@candle.pha.pa.us>; Mon, 3 Nov 1997 09:31:02 -0500 (EST)
Received: from hub.org (hub.org [209.47.148.200]) by renoir.op.net (o1/$ Revision: 1.14 $) with ESMTP id JAA07345 for <maillist@candle.pha.pa.us>; Mon, 3 Nov 1997 09:13:20 -0500 (EST)
Received: from localhost (majordom@localhost) by hub.org (8.8.5/8.7.5) with SMTP id IAA13315; Mon, 3 Nov 1997 08:50:26 -0500 (EST)
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Mon, 03 Nov 1997 08:48:07 -0500 (EST)
Received: (from majordom@localhost) by hub.org (8.8.5/8.7.5) id IAA11722 for pgsql-hackers-outgoing; Mon, 3 Nov 1997 08:48:02 -0500 (EST)
Received: from www.krasnet.ru (www.krasnet.ru [193.125.44.86]) by hub.org (8.8.5/8.7.5) with ESMTP id IAA11539 for <hackers@postgreSQL.org>; Mon, 3 Nov 1997 08:47:34 -0500 (EST)
Received: from www.krasnet.ru (www.krasnet.ru [193.125.44.86]) by www.krasnet.ru (8.8.7/8.7.3) with SMTP id UAA19066; Mon, 3 Nov 1997 20:48:04 +0700 (KRS)
Message-ID: <345DD614.345BF651@sable.krasnoyarsk.su>
Date: Mon, 03 Nov 1997 20:48:04 +0700
From: "Vadim B. Mikheev" <vadim@sable.krasnoyarsk.su>
Organization: ITTS (Krasnoyarsk)
X-Mailer: Mozilla 3.01 (X11; I; FreeBSD 2.2.5-RELEASE i386)
MIME-Version: 1.0
To: Marc Howard Zuckman <marc@fallon.classyad.com>
CC: Bruce Momjian <maillist@candle.pha.pa.us>, hackers@postgreSQL.org
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
References: <Pine.LNX.3.95.971103090709.21917A-100000@fallon.classyad.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Sender: owner-hackers@hub.org
Precedence: bulk
Status: OR
Marc Howard Zuckman wrote:
>
> On Mon, 3 Nov 1997, Bruce Momjian wrote:
>
> > With fsync off, I just did an insert of 1000 integers into a table
> > containing a single int4 column and no indexes, and it completed in 2.3
> > seconds. This is on the new source tree.. That is 434 inserts/second.
> > Pretty major performance, or 2.3 ms/insert. This is on a idle PP200
> > with UltraSCSI drives.
> >
> > With fsync on, the time goes to 51 seconds. Wow, big difference.
>
> If better alternative error recovery methods were available, perhaps
> a facility to replay an interval transactions log from a prior dump,
> it would be reasonable to run the backend without fsync and
> take advantage of the performance gains.
???
>
> I don't know the answer, but I suspect that the commercial databases
> don't "fsync" the way pgsql does.
Could someone try 1000 int4 inserts using postgres and
some commercial database (on the same machine) ?
Vadim
From owner-pgsql-hackers@hub.org Mon Nov 3 09:01:02 1997
Received: from renoir.op.net (root@renoir.op.net [206.84.208.4])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id JAA01183
for <maillist@candle.pha.pa.us>; Mon, 3 Nov 1997 09:01:00 -0500 (EST)
Received: from hub.org (hub.org [209.47.148.200]) by renoir.op.net (o1/$ Revision: 1.14 $) with ESMTP id IAA06632 for <maillist@candle.pha.pa.us>; Mon, 3 Nov 1997 08:51:58 -0500 (EST)
Received: from localhost (majordom@localhost) by hub.org (8.8.5/8.7.5) with SMTP id IAA05964; Mon, 3 Nov 1997 08:39:39 -0500 (EST)
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Mon, 03 Nov 1997 08:37:32 -0500 (EST)
Received: (from majordom@localhost) by hub.org (8.8.5/8.7.5) id IAA04729 for pgsql-hackers-outgoing; Mon, 3 Nov 1997 08:37:26 -0500 (EST)
Received: from fallon.classyad.com (root@classyad.com [152.160.43.1]) by hub.org (8.8.5/8.7.5) with ESMTP id IAA04614 for <hackers@postgreSQL.org>; Mon, 3 Nov 1997 08:37:16 -0500 (EST)
Received: from fallon.classyad.com (marc@fallon.classyad.com [152.160.43.1]) by fallon.classyad.com (8.8.5/8.7.3) with SMTP id JAA22108; Mon, 3 Nov 1997 09:11:09 -0500
Date: Mon, 3 Nov 1997 09:11:09 -0500 (EST)
From: Marc Howard Zuckman <marc@fallon.classyad.com>
To: Bruce Momjian <maillist@candle.pha.pa.us>
cc: "Vadim B. Mikheev" <vadim@sable.krasnoyarsk.su>, hackers@postgreSQL.org
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
In-Reply-To: <199711030513.AAA23474@candle.pha.pa.us>
Message-ID: <Pine.LNX.3.95.971103090709.21917A-100000@fallon.classyad.com>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Sender: owner-hackers@hub.org
Precedence: bulk
Status: OR
On Mon, 3 Nov 1997, Bruce Momjian wrote:
> >
> > Removed...
> >
> > Also, ItemPointerData t_chain (6 bytes) removed from HeapTupleHeader.
> > CommandId is uint32 now (up to the 2^32 - 1 commands per transaction).
> > DOUBLEALIGN(Sizeof(HeapTupleHeader)) is 40 bytes now.
> >
> > 1000 inserts (into table with single int4 column, 1 insert per transaction)
> > takes 70 - 80 sec now (12.5 - 14 transactions/sec).
> > This is hardware/OS limitation:
> >
> > fd = open ("t", O_RDWR);
> > for (i = 1; i <= 1000; i++)
> > {
> > lseek(fd, 0, SEEK_END);
> > write(fd, buf, 56);
> > fsync(fd);
> > }
> > close (fd);
> >
> > takes 33 - 39 sec and so it's not possible to be faster
> > having 2 fsync-s per transaction.
> >
> > The same test on 6.2.1: 92 - 107 sec
>
> With fsync off, I just did an insert of 1000 integers into a table
> containing a single int4 column and no indexes, and it completed in 2.3
> seconds. This is on the new source tree.. That is 434 inserts/second.
> Pretty major performance, or 2.3 ms/insert. This is on a idle PP200
> with UltraSCSI drives.
>
> With fsync on, the time goes to 51 seconds. Wow, big difference.
If better alternative error recovery methods were available, perhaps
a facility to replay an interval transactions log from a prior dump,
it would be reasonable to run the backend without fsync and
take advantage of the performance gains.
I don't know the answer, but I suspect that the commercial databases
don't "fsync" the way pgsql does.
Marc Zuckman
marc@fallon.classyad.com
_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
_ Visit The Home and Condo MarketPlace _
_ http://www.ClassyAd.com _
_ _
_ FREE basic property listings/advertisements and searches. _
_ _
_ Try our premium, yet inexpensive services for a real _
_ selling or buying edge! _
_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
From owner-pgsql-hackers@hub.org Mon Nov 3 11:31:03 1997
Received: from renoir.op.net (root@renoir.op.net [206.84.208.4])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id LAA04080
for <maillist@candle.pha.pa.us>; Mon, 3 Nov 1997 11:31:00 -0500 (EST)
Received: from hub.org (hub.org [209.47.148.200]) by renoir.op.net (o1/$ Revision: 1.14 $) with ESMTP id LAA13680 for <maillist@candle.pha.pa.us>; Mon, 3 Nov 1997 11:21:30 -0500 (EST)
Received: from localhost (majordom@localhost) by hub.org (8.8.5/8.7.5) with SMTP id LAA07566; Mon, 3 Nov 1997 11:04:52 -0500 (EST)
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Mon, 03 Nov 1997 11:02:59 -0500 (EST)
Received: (from majordom@localhost) by hub.org (8.8.5/8.7.5) id LAA07372 for pgsql-hackers-outgoing; Mon, 3 Nov 1997 11:02:52 -0500 (EST)
Received: from candle.pha.pa.us (root@s3-03.ppp.op.net [206.84.210.195]) by hub.org (8.8.5/8.7.5) with ESMTP id LAA07196 for <hackers@postgreSQL.org>; Mon, 3 Nov 1997 11:02:22 -0500 (EST)
Received: (from maillist@localhost)
by candle.pha.pa.us (8.8.5/8.8.5) id KAA02525;
Mon, 3 Nov 1997 10:42:03 -0500 (EST)
From: Bruce Momjian <maillist@candle.pha.pa.us>
Message-Id: <199711031542.KAA02525@candle.pha.pa.us>
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
To: vadim@sable.krasnoyarsk.su (Vadim B. Mikheev)
Date: Mon, 3 Nov 1997 10:42:03 -0500 (EST)
Cc: marc@fallon.classyad.com, hackers@postgreSQL.org
In-Reply-To: <345DD614.345BF651@sable.krasnoyarsk.su> from "Vadim B. Mikheev" at Nov 3, 97 08:48:04 pm
X-Mailer: ELM [version 2.4 PL25]
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Sender: owner-hackers@hub.org
Precedence: bulk
Status: OR
> > I don't know the answer, but I suspect that the commercial databases
> > don't "fsync" the way pgsql does.
>
> Could someone try 1000 int4 inserts using postgres and
> some commercial database (on the same machine) ?
I have been thinking about this since seeing the performance change
with/without fsync.
Commerical databases usually do a log write every 5 or 15 minutes, and
guarantee the logs will contain everything up to this time interval.
Couldn't we have some such mechanism? Usually they have raw space, so
they can control when the data is hitting the disk. Using a file
system, some of it may be getting to the disk without our knowing it.
What exactly is a scenario where lack of doing explicit fsync's will
cause data corruption, rather than just lost data from the past few
minutes?
I think Vadim has gotten fsync's down to fsync'ing the modified data
page, and pg_log.
Let's suppose we did not fsync. There could be cases where pg_log was
fsync'ed by the OS, and some of the modified data pages are fyncs'ed by
the OS, but not others. This would leave us with a partial transaction.
However, let's suppose we prevent pg_log from being fsync'ed somehow.
Then, because we have a no-overwrite database, we could keep control of
this, and write of some data pages, but not others would not cause us
problems because the pg_log would show all such transactions, which had
not had all their modified data pages fsync'ed, as non-committed.
Perhaps we can even set a flag in pg_log every five minutes to indicate
whether all buffers for the page have been flushed? That way we could
not have to worry about preventing flushing of pg_log.
Comments?
--
Bruce Momjian
maillist@candle.pha.pa.us
From owner-pgsql-hackers@hub.org Mon Nov 3 12:00:42 1997
Received: from hub.org (hub.org [209.47.148.200])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id MAA04456
for <maillist@candle.pha.pa.us>; Mon, 3 Nov 1997 12:00:40 -0500 (EST)
Received: from localhost (majordom@localhost) by hub.org (8.8.5/8.7.5) with SMTP id LAA26054; Mon, 3 Nov 1997 11:46:49 -0500 (EST)
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Mon, 03 Nov 1997 11:46:33 -0500 (EST)
Received: (from majordom@localhost) by hub.org (8.8.5/8.7.5) id LAA25932 for pgsql-hackers-outgoing; Mon, 3 Nov 1997 11:46:30 -0500 (EST)
Received: from orion.SAPserv.Hamburg.dsh.de (polaris.sapserv.debis.de [53.2.131.8]) by hub.org (8.8.5/8.7.5) with SMTP id LAA25750 for <hackers@postgreSQL.org>; Mon, 3 Nov 1997 11:45:53 -0500 (EST)
Received: by orion.SAPserv.Hamburg.dsh.de
(Linux Smail3.1.29.1 #1)}
id m0xSPfE-000BGZC; Mon, 3 Nov 97 17:47 MET
Message-Id: <m0xSPfE-000BGZC@orion.SAPserv.Hamburg.dsh.de>
From: wieck@sapserv.debis.de
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
To: maillist@candle.pha.pa.us (Bruce Momjian)
Date: Mon, 3 Nov 1997 17:47:43 +0100 (MET)
Cc: vadim@sable.krasnoyarsk.su, marc@fallon.classyad.com,
hackers@postgreSQL.org
Reply-To: wieck@sapserv.debis.de (Jan Wieck)
In-Reply-To: <199711031542.KAA02525@candle.pha.pa.us> from "Bruce Momjian" at Nov 3, 97 10:42:03 am
X-Mailer: ELM [version 2.4 PL25]
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: 8bit
Sender: owner-hackers@hub.org
Precedence: bulk
Status: OR
>
> > > I don't know the answer, but I suspect that the commercial databases
> > > don't "fsync" the way pgsql does.
> >
> > Could someone try 1000 int4 inserts using postgres and
> > some commercial database (on the same machine) ?
>
> I have been thinking about this since seeing the performance change
> with/without fsync.
>
> Commerical databases usually do a log write every 5 or 15 minutes, and
> guarantee the logs will contain everything up to this time interval.
>
Without fsync PostgreSQL would only loose data if the OS
crashes between the last write operation of a backend and the
next regular update sync. This is seldom but if it happens it
really hurts.
A database can omit fsync on data files (e.g. tablespaces) if
it writes a redo log. With that redo log, a backup can be
restored and than all transactions since the backup redone.
PostgreSQL doesn't write such a redo log. So an OS crash
after the fsync of pg_log could corrupt the database without
a chance to recover.
Isn't it time to get an (optional) redo log. I don't exactly
know all the places where our datafiles can get modified, but
I hope this is only done in the heap access methods and
vacuum. So these are the places from where the redo log data
comes from (plus transaction commit/rollback).
Until later, Jan
--
#define OPINIONS "they are all mine - not those of debis or daimler-benz"
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================== wieck@sapserv.debis.de (Jan Wieck) #
From owner-pgsql-hackers@hub.org Mon Nov 3 14:01:06 1997
Received: from renoir.op.net (root@renoir.op.net [206.84.208.4])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id OAA06775
for <maillist@candle.pha.pa.us>; Mon, 3 Nov 1997 14:01:04 -0500 (EST)
Received: from hub.org (hub.org [209.47.148.200]) by renoir.op.net (o1/$ Revision: 1.14 $) with ESMTP id NAA22235 for <maillist@candle.pha.pa.us>; Mon, 3 Nov 1997 13:43:15 -0500 (EST)
Received: from localhost (majordom@localhost) by hub.org (8.8.5/8.7.5) with SMTP id NAA11482; Mon, 3 Nov 1997 13:32:40 -0500 (EST)
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Mon, 03 Nov 1997 13:32:02 -0500 (EST)
Received: (from majordom@localhost) by hub.org (8.8.5/8.7.5) id NAA11204 for pgsql-hackers-outgoing; Mon, 3 Nov 1997 13:31:58 -0500 (EST)
Received: from candle.pha.pa.us (root@s3-03.ppp.op.net [206.84.210.195]) by hub.org (8.8.5/8.7.5) with ESMTP id NAA11119 for <hackers@postgreSQL.org>; Mon, 3 Nov 1997 13:31:44 -0500 (EST)
Received: (from maillist@localhost)
by candle.pha.pa.us (8.8.5/8.8.5) id MAA05464;
Mon, 3 Nov 1997 12:59:01 -0500 (EST)
From: Bruce Momjian <maillist@candle.pha.pa.us>
Message-Id: <199711031759.MAA05464@candle.pha.pa.us>
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
To: wieck@sapserv.debis.de
Date: Mon, 3 Nov 1997 12:59:01 -0500 (EST)
Cc: vadim@sable.krasnoyarsk.su, marc@fallon.classyad.com,
hackers@postgreSQL.org
In-Reply-To: <m0xSPfE-000BGZC@orion.SAPserv.Hamburg.dsh.de> from "wieck@sapserv.debis.de" at Nov 3, 97 05:47:43 pm
X-Mailer: ELM [version 2.4 PL25]
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Sender: owner-hackers@hub.org
Precedence: bulk
Status: OR
>
> >
> > > > I don't know the answer, but I suspect that the commercial databases
> > > > don't "fsync" the way pgsql does.
> > >
> > > Could someone try 1000 int4 inserts using postgres and
> > > some commercial database (on the same machine) ?
> >
> > I have been thinking about this since seeing the performance change
> > with/without fsync.
> >
> > Commerical databases usually do a log write every 5 or 15 minutes, and
> > guarantee the logs will contain everything up to this time interval.
> >
>
> Without fsync PostgreSQL would only loose data if the OS
> crashes between the last write operation of a backend and the
> next regular update sync. This is seldom but if it happens it
> really hurts.
>
> A database can omit fsync on data files (e.g. tablespaces) if
> it writes a redo log. With that redo log, a backup can be
> restored and than all transactions since the backup redone.
>
> PostgreSQL doesn't write such a redo log. So an OS crash
> after the fsync of pg_log could corrupt the database without
> a chance to recover.
>
> Isn't it time to get an (optional) redo log. I don't exactly
> know all the places where our datafiles can get modified, but
> I hope this is only done in the heap access methods and
> vacuum. So these are the places from where the redo log data
> comes from (plus transaction commit/rollback).
>
Yes, but because we are a non-over-write database, I don't see why we
can't just do this without a redo log.
Every five minutes, we fsync() all dirty pages, mark all completed
transactions as fsync'ed in pg_log, and fsync() pg_log.
On postmaster startup, any transaction marked as completed, but not
marked as fsync'ed gets marked as aborted.
Of course, all vacuum operations would have to be fsync'ed.
Comments?
--
Bruce Momjian
maillist@candle.pha.pa.us
From owner-pgsql-hackers@hub.org Mon Nov 3 16:46:01 1997
Received: from renoir.op.net (root@renoir.op.net [206.84.208.4])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id QAA10292
for <maillist@candle.pha.pa.us>; Mon, 3 Nov 1997 16:45:59 -0500 (EST)
Received: from hub.org (hub.org [209.47.148.200]) by renoir.op.net (o1/$ Revision: 1.14 $) with ESMTP id QAA02040 for <maillist@candle.pha.pa.us>; Mon, 3 Nov 1997 16:42:40 -0500 (EST)
Received: from localhost (majordom@localhost) by hub.org (8.8.5/8.7.5) with SMTP id QAA17422; Mon, 3 Nov 1997 16:34:28 -0500 (EST)
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Mon, 03 Nov 1997 16:34:10 -0500 (EST)
Received: (from majordom@localhost) by hub.org (8.8.5/8.7.5) id QAA17210 for pgsql-hackers-outgoing; Mon, 3 Nov 1997 16:34:06 -0500 (EST)
Received: from fallon.classyad.com (root@classyad.com [152.160.43.1]) by hub.org (8.8.5/8.7.5) with ESMTP id QAA16690 for <hackers@postgreSQL.org>; Mon, 3 Nov 1997 16:33:27 -0500 (EST)
Received: from fallon.classyad.com (marc@fallon.classyad.com [152.160.43.1]) by fallon.classyad.com (8.8.5/8.7.3) with SMTP id RAA32498; Mon, 3 Nov 1997 17:33:42 -0500
Date: Mon, 3 Nov 1997 17:33:42 -0500 (EST)
From: Marc Howard Zuckman <marc@fallon.classyad.com>
To: Bruce Momjian <maillist@candle.pha.pa.us>
cc: wieck@sapserv.debis.de, vadim@sable.krasnoyarsk.su, hackers@postgreSQL.org
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
In-Reply-To: <199711031759.MAA05464@candle.pha.pa.us>
Message-ID: <Pine.LNX.3.95.971103173129.32055B-100000@fallon.classyad.com>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Sender: owner-hackers@hub.org
Precedence: bulk
Status: OR
On Mon, 3 Nov 1997, Bruce Momjian wrote:
> >
> > >
> > > > > I don't know the answer, but I suspect that the commercial databases
> > > > > don't "fsync" the way pgsql does.
> > > >
> > > > Could someone try 1000 int4 inserts using postgres and
> > > > some commercial database (on the same machine) ?
> > >
> > > I have been thinking about this since seeing the performance change
> > > with/without fsync.
> > >
> > > Commerical databases usually do a log write every 5 or 15 minutes, and
> > > guarantee the logs will contain everything up to this time interval.
> > >
> >
> > Without fsync PostgreSQL would only loose data if the OS
> > crashes between the last write operation of a backend and the
> > next regular update sync. This is seldom but if it happens it
> > really hurts.
> >
> > A database can omit fsync on data files (e.g. tablespaces) if
> > it writes a redo log. With that redo log, a backup can be
> > restored and than all transactions since the backup redone.
> >
> > PostgreSQL doesn't write such a redo log. So an OS crash
> > after the fsync of pg_log could corrupt the database without
> > a chance to recover.
> >
> > Isn't it time to get an (optional) redo log. I don't exactly
> > know all the places where our datafiles can get modified, but
> > I hope this is only done in the heap access methods and
> > vacuum. So these are the places from where the redo log data
> > comes from (plus transaction commit/rollback).
> >
>
> Yes, but because we are a non-over-write database, I don't see why we
> can't just do this without a redo log.
Because if the hard drive is the reason for the failure (instead of
power out, OS bites dust, etc), the database won't be of much help.
The redo log should be on a device different than the database.
Marc Zuckman
marc@fallon.classyad.com
_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
_ Visit The Home and Condo MarketPlace _
_ http://www.ClassyAd.com _
_ _
_ FREE basic property listings/advertisements and searches. _
_ _
_ Try our premium, yet inexpensive services for a real _
_ selling or buying edge! _
_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
From maillist Mon Nov 3 22:59:31 1997
Received: (from maillist@localhost)
by candle.pha.pa.us (8.8.5/8.8.5) id WAA16264;
Mon, 3 Nov 1997 22:59:31 -0500 (EST)
From: Bruce Momjian <maillist>
Message-Id: <199711040359.WAA16264@candle.pha.pa.us>
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
To: maillist@candle.pha.pa.us (Bruce Momjian)
Date: Mon, 3 Nov 1997 22:59:30 -0500 (EST)
Cc: vadim@sable.krasnoyarsk.su, marc@fallon.classyad.com,
hackers@postgreSQL.org
In-Reply-To: <199711031542.KAA02525@candle.pha.pa.us> from "Bruce Momjian" at Nov 3, 97 10:42:03 am
X-Mailer: ELM [version 2.4 PL25]
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Status: OR
>
> > > I don't know the answer, but I suspect that the commercial databases
> > > don't "fsync" the way pgsql does.
> >
> > Could someone try 1000 int4 inserts using postgres and
> > some commercial database (on the same machine) ?
>
> I have been thinking about this since seeing the performance change
> with/without fsync.
>
> Commercial databases usually do a log write every 5 or 15 minutes, and
> guarantee the logs will contain everything up to this time interval.
>
> Couldn't we have some such mechanism? Usually they have raw space, so
> they can control when the data is hitting the disk. Using a file
> system, some of it may be getting to the disk without our knowing it.
>
> What exactly is a scenario where lack of doing explicit fsync's will
> cause data corruption, rather than just lost data from the past few
> minutes?
>
> I think Vadim has gotten fsync's down to fsync'ing the modified data
> page, and pg_log.
>
> Let's suppose we did not fsync. There could be cases where pg_log was
> fsync'ed by the OS, and some of the modified data pages are fyncs'ed by
> the OS, but not others. This would leave us with a partial transaction.
>
> However, let's suppose we prevent pg_log from being fsync'ed somehow.
> Then, because we have a no-overwrite database, we could keep control of
> this, and write of some data pages, but not others would not cause us
> problems because the pg_log would show all such transactions, which had
> not had all their modified data pages fsync'ed, as non-committed.
>
> Perhaps we can even set a flag in pg_log every five minutes to indicate
> whether all buffers for the page have been flushed? That way we could
> not have to worry about preventing flushing of pg_log.
>
> Comments?
OK, here is a more formal description of what I am suggesting. It will
give us commercial dbms reliability with no-fsync performance.
Commercial dbms's usually only give restore up to 5 minutes before the
crash, and this is what I am suggesting. If we can do this, we can
remove the no-fsync option.
First, lets suppose there exists a shared queue that is visible to all
backends and the postmaster that allows transaction id's to be added to
the queue. We also add a bit to the pg_log record called 'been_synced'
that is initially false.
OK, once a backend starts a transaction, it puts a transaction id in
pg_log. Once the transaction is finished, it is marked as committed.
At the same time, we now put the transaction id on the shared queue.
Every five minutes, or as defined by the administrator, the postmaster
does a sync() call. On my OS, anyone use can call sync, and I think
this is typical. update/pagecleaner does this every 30 seconds anyway,
so it is no big deal for the postmaster to call it every 5 minutes. The
nice thing about this is that the OS does the syncing of all the dirty
pages for us. (An alarm() call can set up this 5 minute timing.)
The postmaster then locks the shared transaction id queue, makes a copy
of the entries in the queue, clears the queue, and unlocks the queue.
It does this so no one else modifies the queue while it is being
cleared.
The postmaster then goes through pg_log, and marks each transaction as
'been_synced'.
The postmaster also performs this on shutdown.
On postmaster startup, all transactions are checked and any transaction
that is marked as committed but not 'been_synced' is marked as not
committed. In this way, we prevent non-synced or partially synced
transactions from being used.
Of course, vacuum would have to do normal fsyncs because it is removing
the transaction log.
We need the shared transaction id queue because there is no way to find
the newly committed transactions since the last sync. A transaction
can last for hours.
--
Bruce Momjian
maillist@candle.pha.pa.us
From owner-pgsql-hackers@hub.org Tue Nov 4 02:13:08 1997
Received: from hub.org (hub.org [209.47.148.200])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id CAA17544
for <maillist@candle.pha.pa.us>; Tue, 4 Nov 1997 02:13:06 -0500 (EST)
Received: from localhost (majordom@localhost) by hub.org (8.8.5/8.7.5) with SMTP id CAA14126; Tue, 4 Nov 1997 02:07:55 -0500 (EST)
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Tue, 04 Nov 1997 02:04:59 -0500 (EST)
Received: (from majordom@localhost) by hub.org (8.8.5/8.7.5) id CAA12859 for pgsql-hackers-outgoing; Tue, 4 Nov 1997 02:04:51 -0500 (EST)
Received: from orion.SAPserv.Hamburg.dsh.de (polaris.sapserv.debis.de [53.2.131.8]) by hub.org (8.8.5/8.7.5) with SMTP id CAA12625 for <hackers@postgreSQL.org>; Tue, 4 Nov 1997 02:04:12 -0500 (EST)
Received: by orion.SAPserv.Hamburg.dsh.de
(Linux Smail3.1.29.1 #1)}
id m0xSd44-000BFQC; Tue, 4 Nov 97 08:06 MET
Message-Id: <m0xSd44-000BFQC@orion.SAPserv.Hamburg.dsh.de>
From: wieck@sapserv.debis.de
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
To: maillist@candle.pha.pa.us (Bruce Momjian)
Date: Tue, 4 Nov 1997 08:06:16 +0100 (MET)
Cc: maillist@candle.pha.pa.us, vadim@sable.krasnoyarsk.su,
marc@fallon.classyad.com, hackers@postgreSQL.org
Reply-To: wieck@sapserv.debis.de (Jan Wieck)
In-Reply-To: <199711040359.WAA16264@candle.pha.pa.us> from "Bruce Momjian" at Nov 3, 97 10:59:30 pm
X-Mailer: ELM [version 2.4 PL25]
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: 8bit
Sender: owner-hackers@hub.org
Precedence: bulk
Status: OR
> OK, here is a more formal description of what I am suggesting. It will
> give us commercial dbms reliability with no-fsync performance.
> Commercial dbms's usually only give restore up to 5 minutes before the
> crash, and this is what I am suggesting. If we can do this, we can
> remove the no-fsync option.
I'm not 100% sure but as far as I know Oracle, it can recover
up to the last committed transaction using the online redo
logs. And even if commercial dbms's aren't able to do that,
it should be our target.
> [description about transaction queue]
This all depends on the fact that PostgreSQL is a no
overwrite dbms. Otherwise the space of deleted tuples might
get overwritten by later transactions and the information is
finally lost.
Another issue: All we up to now though of are crashes where
the database files are still usable after restart. But take
the simple case of a write error. A new bad block or track
will get remapped (in some way) but the data in it is lost.
So we end up with one or more totally corrupted database
files. And I don't trust mirrored disks farer than I can
throw them. A bug in the OS or a memory failure (many new
PeeCee boards don't support parity and even with parity a two
bit failure is still the wrong data but with a valid parity
bit) can also currupt the data.
I still prefer redo logs. They should reside on a different
disk and the possibility of loosing the database files along
with the redo log is very small.
Until later, Jan
--
#define OPINIONS "they are all mine - not those of debis or daimler-benz"
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================== wieck@sapserv.debis.de (Jan Wieck) #
From vadim@sable.krasnoyarsk.su Tue Nov 4 04:12:50 1997
Received: from renoir.op.net (root@renoir.op.net [206.84.208.4])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id EAA18487
for <maillist@candle.pha.pa.us>; Tue, 4 Nov 1997 04:12:48 -0500 (EST)
Received: from www.krasnet.ru (www.krasnet.ru [193.125.44.86]) by renoir.op.net (o1/$ Revision: 1.14 $) with ESMTP id EAA03152 for <maillist@candle.pha.pa.us>; Tue, 4 Nov 1997 04:12:06 -0500 (EST)
Received: from www.krasnet.ru (www.krasnet.ru [193.125.44.86]) by www.krasnet.ru (8.8.7/8.7.3) with SMTP id QAA20591; Tue, 4 Nov 1997 16:14:06 +0700 (KRS)
Sender: root@www.krasnet.ru
Message-ID: <345EE75D.398A68D@sable.krasnoyarsk.su>
Date: Tue, 04 Nov 1997 16:14:05 +0700
From: "Vadim B. Mikheev" <vadim@sable.krasnoyarsk.su>
Organization: ITTS (Krasnoyarsk)
X-Mailer: Mozilla 3.01 (X11; I; FreeBSD 2.2.5-RELEASE i386)
MIME-Version: 1.0
To: Bruce Momjian <maillist@candle.pha.pa.us>
CC: marc@fallon.classyad.com, hackers@postgreSQL.org
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
References: <199711040359.WAA16264@candle.pha.pa.us>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Status: OR
Bruce Momjian wrote:
>
> OK, here is a more formal description of what I am suggesting. It will
> give us commercial dbms reliability with no-fsync performance.
> Commercial dbms's usually only give restore up to 5 minutes before the
^^^^^^^^^^^^^^^^^^^^^^^
I'm sure that this is not true!
If on-line redo_file is damaged then you have
single ability: restore your last backup.
In all other cases database will be recovered up to the last
committed transaction automatically!
DBMS-s using WAL have to fsync only redo file on commit
(and they do it!), non-overwriting systems have to
fsync data files and transaction log.
We could optimize fsync-s for multi-user environment: do not
fsync when we're ensured that our changes flushed to disk by
another backend.
> crash, and this is what I am suggesting. If we can do this, we can
> remove the no-fsync option.
>
...
>
> On postmaster startup, all transactions are checked and any transaction
> that is marked as committed but not 'been_synced' is marked as not
> committed. In this way, we prevent non-synced or partially synced
> transactions from being used.
And what should users (ensured that their transaction are
committed) do in this case ?
Vadim
From owner-pgsql-hackers@hub.org Tue Nov 4 04:21:04 1997
Received: from hub.org (hub.org [209.47.148.200])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id EAA18536
for <maillist@candle.pha.pa.us>; Tue, 4 Nov 1997 04:21:01 -0500 (EST)
Received: from localhost (majordom@localhost) by hub.org (8.8.5/8.7.5) with SMTP id EAA15551; Tue, 4 Nov 1997 04:15:15 -0500 (EST)
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Tue, 04 Nov 1997 04:14:23 -0500 (EST)
Received: (from majordom@localhost) by hub.org (8.8.5/8.7.5) id EAA14464 for pgsql-hackers-outgoing; Tue, 4 Nov 1997 04:14:18 -0500 (EST)
Received: from www.krasnet.ru (www.krasnet.ru [193.125.44.86]) by hub.org (8.8.5/8.7.5) with ESMTP id EAA13437 for <hackers@postgreSQL.org>; Tue, 4 Nov 1997 04:13:33 -0500 (EST)
Received: from www.krasnet.ru (www.krasnet.ru [193.125.44.86]) by www.krasnet.ru (8.8.7/8.7.3) with SMTP id QAA20591; Tue, 4 Nov 1997 16:14:06 +0700 (KRS)
Message-ID: <345EE75D.398A68D@sable.krasnoyarsk.su>
Date: Tue, 04 Nov 1997 16:14:05 +0700
From: "Vadim B. Mikheev" <vadim@sable.krasnoyarsk.su>
Organization: ITTS (Krasnoyarsk)
X-Mailer: Mozilla 3.01 (X11; I; FreeBSD 2.2.5-RELEASE i386)
MIME-Version: 1.0
To: Bruce Momjian <maillist@candle.pha.pa.us>
CC: marc@fallon.classyad.com, hackers@postgreSQL.org
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
References: <199711040359.WAA16264@candle.pha.pa.us>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Sender: owner-hackers@hub.org
Precedence: bulk
Status: OR
Bruce Momjian wrote:
>
> OK, here is a more formal description of what I am suggesting. It will
> give us commercial dbms reliability with no-fsync performance.
> Commercial dbms's usually only give restore up to 5 minutes before the
^^^^^^^^^^^^^^^^^^^^^^^
I'm sure that this is not true!
If on-line redo_file is damaged then you have
single ability: restore your last backup.
In all other cases database will be recovered up to the last
committed transaction automatically!
DBMS-s using WAL have to fsync only redo file on commit
(and they do it!), non-overwriting systems have to
fsync data files and transaction log.
We could optimize fsync-s for multi-user environment: do not
fsync when we're ensured that our changes flushed to disk by
another backend.
> crash, and this is what I am suggesting. If we can do this, we can
> remove the no-fsync option.
>
...
>
> On postmaster startup, all transactions are checked and any transaction
> that is marked as committed but not 'been_synced' is marked as not
> committed. In this way, we prevent non-synced or partially synced
> transactions from being used.
And what should users (ensured that their transaction are
committed) do in this case ?
Vadim
From owner-pgsql-hackers@hub.org Tue Nov 4 06:43:00 1997
Received: from hub.org (hub.org [209.47.148.200])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id GAA19743
for <maillist@candle.pha.pa.us>; Tue, 4 Nov 1997 06:42:57 -0500 (EST)
Received: from localhost (majordom@localhost) by hub.org (8.8.5/8.7.5) with SMTP id GAA10352; Tue, 4 Nov 1997 06:36:08 -0500 (EST)
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Tue, 04 Nov 1997 06:35:42 -0500 (EST)
Received: (from majordom@localhost) by hub.org (8.8.5/8.7.5) id GAA10158 for pgsql-hackers-outgoing; Tue, 4 Nov 1997 06:35:37 -0500 (EST)
Received: from candle.pha.pa.us (maillist@s3-03.ppp.op.net [206.84.210.195]) by hub.org (8.8.5/8.7.5) with ESMTP id GAA10096 for <hackers@postgreSQL.org>; Tue, 4 Nov 1997 06:35:27 -0500 (EST)
Received: (from maillist@localhost)
by candle.pha.pa.us (8.8.5/8.8.5) id GAA19665;
Tue, 4 Nov 1997 06:35:10 -0500 (EST)
From: Bruce Momjian <maillist@candle.pha.pa.us>
Message-Id: <199711041135.GAA19665@candle.pha.pa.us>
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
To: wieck@sapserv.debis.de
Date: Tue, 4 Nov 1997 06:35:10 -0500 (EST)
Cc: hackers@postgreSQL.org (PostgreSQL-development)
In-Reply-To: <m0xSd44-000BFQC@orion.SAPserv.Hamburg.dsh.de> from "wieck@sapserv.debis.de" at Nov 4, 97 08:06:16 am
X-Mailer: ELM [version 2.4 PL25]
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Sender: owner-hackers@hub.org
Precedence: bulk
Status: OR
>
> > OK, here is a more formal description of what I am suggesting. It will
> > give us commercial dbms reliability with no-fsync performance.
> > Commercial dbms's usually only give restore up to 5 minutes before the
> > crash, and this is what I am suggesting. If we can do this, we can
> > remove the no-fsync option.
>
> I'm not 100% sure but as far as I know Oracle, it can recover
> up to the last committed transaction using the online redo
> logs. And even if commercial dbms's aren't able to do that,
> it should be our target.
>
> > [description about transaction queue]
>
> This all depends on the fact that PostgreSQL is a no
> overwrite dbms. Otherwise the space of deleted tuples might
> get overwritten by later transactions and the information is
> finally lost.
>
> Another issue: All we up to now though of are crashes where
> the database files are still usable after restart. But take
> the simple case of a write error. A new bad block or track
> will get remapped (in some way) but the data in it is lost.
> So we end up with one or more totally corrupted database
> files. And I don't trust mirrored disks farer than I can
> throw them. A bug in the OS or a memory failure (many new
> PeeCee boards don't support parity and even with parity a two
> bit failure is still the wrong data but with a valid parity
> bit) can also currupt the data.
>
> I still prefer redo logs. They should reside on a different
> disk and the possibility of loosing the database files along
> with the redo log is very small.
I have been thinking about re-do logs, and I think it is a good idea.
It would not be hard to have the queries spit out to a separate file
configurable by the user.
--
Bruce Momjian
maillist@candle.pha.pa.us
From owner-pgsql-hackers@hub.org Tue Nov 4 07:31:01 1997
Received: from renoir.op.net (root@renoir.op.net [206.84.208.4])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id HAA22051
for <maillist@candle.pha.pa.us>; Tue, 4 Nov 1997 07:30:59 -0500 (EST)
Received: from hub.org (hub.org [209.47.148.200]) by renoir.op.net (o1/$ Revision: 1.14 $) with ESMTP id HAA07444 for <maillist@candle.pha.pa.us>; Tue, 4 Nov 1997 07:25:14 -0500 (EST)
Received: from localhost (majordom@localhost) by hub.org (8.8.5/8.7.5) with SMTP id HAA08818; Tue, 4 Nov 1997 07:03:30 -0500 (EST)
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Tue, 04 Nov 1997 07:02:44 -0500 (EST)
Received: (from majordom@localhost) by hub.org (8.8.5/8.7.5) id HAA08418 for pgsql-hackers-outgoing; Tue, 4 Nov 1997 07:02:29 -0500 (EST)
Received: from candle.pha.pa.us (root@s3-03.ppp.op.net [206.84.210.195]) by hub.org (8.8.5/8.7.5) with ESMTP id HAA08331 for <hackers@postgreSQL.org>; Tue, 4 Nov 1997 07:02:07 -0500 (EST)
Received: (from maillist@localhost)
by candle.pha.pa.us (8.8.5/8.8.5) id GAA21484;
Tue, 4 Nov 1997 06:50:24 -0500 (EST)
From: Bruce Momjian <maillist@candle.pha.pa.us>
Message-Id: <199711041150.GAA21484@candle.pha.pa.us>
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
To: vadim@sable.krasnoyarsk.su (Vadim B. Mikheev)
Date: Tue, 4 Nov 1997 06:50:24 -0500 (EST)
Cc: marc@fallon.classyad.com, hackers@postgreSQL.org
In-Reply-To: <345EE75D.398A68D@sable.krasnoyarsk.su> from "Vadim B. Mikheev" at Nov 4, 97 04:14:05 pm
X-Mailer: ELM [version 2.4 PL25]
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Sender: owner-hackers@hub.org
Precedence: bulk
Status: OR
>
> Bruce Momjian wrote:
> >
> > OK, here is a more formal description of what I am suggesting. It will
> > give us commercial dbms reliability with no-fsync performance.
> > Commercial dbms's usually only give restore up to 5 minutes before the
> ^^^^^^^^^^^^^^^^^^^^^^^
> I'm sure that this is not true!
You may be right. This five minute figure is when you restore from your
previous backup, then restore from the log file.
Can't we do something like sync every 5 seconds, rather than after every
transaction? It just seems like such overkill.
Actually, I found a problem with my description. Because pg_log is not
fsync'ed, after a crash, pages with new transactions could have been
flushed to disk, but not the pg_log table that contains the transaction
ids. The problem is that the new backend could assign a transaction id
that is already in use.
We could set a flag upon successful shutdown, and if it is not set on
reboot, either do a vacuum to find the max transaction id, and
invalidate all them not in pg_log as synced, or increase the next
transaction id to some huge number and invalidate all them in between.
> If on-line redo_file is damaged then you have
> single ability: restore your last backup.
> In all other cases database will be recovered up to the last
> committed transaction automatically!
>
> DBMS-s using WAL have to fsync only redo file on commit
> (and they do it!), non-overwriting systems have to
> fsync data files and transaction log.
>
> We could optimize fsync-s for multi-user environment: do not
> fsync when we're ensured that our changes flushed to disk by
> another backend.
>
> > crash, and this is what I am suggesting. If we can do this, we can
> > remove the no-fsync option.
> >
> ...
> >
> > On postmaster startup, all transactions are checked and any transaction
> > that is marked as committed but not 'been_synced' is marked as not
> > committed. In this way, we prevent non-synced or partially synced
> > transactions from being used.
>
> And what should users (ensured that their transaction are
> committed) do in this case ?
>
> Vadim
>
>
--
Bruce Momjian
maillist@candle.pha.pa.us
From wieck@sapserv.debis.de Tue Nov 4 07:01:00 1997
Received: from renoir.op.net (root@renoir.op.net [206.84.208.4])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id HAA21697
for <maillist@candle.pha.pa.us>; Tue, 4 Nov 1997 07:00:58 -0500 (EST)
From: wieck@sapserv.debis.de
Received: from orion.SAPserv.Hamburg.dsh.de (polaris.sapserv.debis.de [53.2.131.8]) by renoir.op.net (o1/$ Revision: 1.14 $) with SMTP id GAA06401 for <maillist@candle.pha.pa.us>; Tue, 4 Nov 1997 06:48:25 -0500 (EST)
Received: by orion.SAPserv.Hamburg.dsh.de
(Linux Smail3.1.29.1 #1)}
id m0xShVQ-000BGZC; Tue, 4 Nov 97 12:50 MET
Message-Id: <m0xShVQ-000BGZC@orion.SAPserv.Hamburg.dsh.de>
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
To: maillist@candle.pha.pa.us (Bruce Momjian)
Date: Tue, 4 Nov 1997 12:50:45 +0100 (MET)
Cc: wieck@sapserv.debis.de, hackers@postgreSQL.org
Reply-To: wieck@sapserv.debis.de (Jan Wieck)
In-Reply-To: <199711041135.GAA19665@candle.pha.pa.us> from "Bruce Momjian" at Nov 4, 97 06:35:10 am
X-Mailer: ELM [version 2.4 PL25]
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: 8bit
Status: OR
Bruce Momjian wrote:
> I have been thinking about re-do logs, and I think it is a good idea.
> It would not be hard to have the queries spit out to a separate file
> configurable by the user.
This way the recovery process will be very complicated. When
multiple backends run concurrently, there are multiple
transactions active at the same time. And what tuples are
affected by an update e.g. depends much on the timing.
I had something different in mind. The redo log contains the
information from the executor (e.g. the transactionId, the
tupleId and the new tuple values when calling ExecReplace())
and the information which transactions commit and which not.
When recovering, those operations where the transactions
committed are again passed to the executors functions that do
the real updates with the values from the logfile.
Until later, Jan
--
#define OPINIONS "they are all mine - not those of debis or daimler-benz"
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================== wieck@sapserv.debis.de (Jan Wieck) #
From owner-pgsql-hackers@hub.org Tue Nov 4 07:30:59 1997
Received: from renoir.op.net (root@renoir.op.net [206.84.208.4])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id HAA22048
for <maillist@candle.pha.pa.us>; Tue, 4 Nov 1997 07:30:57 -0500 (EST)
Received: from hub.org (hub.org [209.47.148.200]) by renoir.op.net (o1/$ Revision: 1.14 $) with ESMTP id HAA07189 for <maillist@candle.pha.pa.us>; Tue, 4 Nov 1997 07:18:02 -0500 (EST)
Received: from localhost (majordom@localhost) by hub.org (8.8.5/8.7.5) with SMTP id HAA08856; Tue, 4 Nov 1997 07:03:37 -0500 (EST)
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Tue, 04 Nov 1997 07:03:03 -0500 (EST)
Received: (from majordom@localhost) by hub.org (8.8.5/8.7.5) id HAA08487 for pgsql-hackers-outgoing; Tue, 4 Nov 1997 07:02:46 -0500 (EST)
Received: from candle.pha.pa.us (maillist@s3-03.ppp.op.net [206.84.210.195]) by hub.org (8.8.5/8.7.5) with ESMTP id HAA08192 for <hackers@postgreSQL.org>; Tue, 4 Nov 1997 07:02:02 -0500 (EST)
Received: (from maillist@localhost)
by candle.pha.pa.us (8.8.5/8.8.5) id HAA21653;
Tue, 4 Nov 1997 07:00:20 -0500 (EST)
From: Bruce Momjian <maillist@candle.pha.pa.us>
Message-Id: <199711041200.HAA21653@candle.pha.pa.us>
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!u
To: vadim@sable.krasnoyarsk.su (Vadim B. Mikheev)
Date: Tue, 4 Nov 1997 07:00:19 -0500 (EST)
Cc: marc@fallon.classyad.com, hackers@postgreSQL.org
In-Reply-To: <345EE75D.398A68D@sable.krasnoyarsk.su> from "Vadim B. Mikheev" at Nov 4, 97 04:14:05 pm
X-Mailer: ELM [version 2.4 PL25]
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Sender: owner-hackers@hub.org
Precedence: bulk
Status: OR
>
> Bruce Momjian wrote:
> >
> > OK, here is a more formal description of what I am suggesting. It will
> > give us commercial dbms reliability with no-fsync performance.
> > Commercial dbms's usually only give restore up to 5 minutes before the
> ^^^^^^^^^^^^^^^^^^^^^^^
> I'm sure that this is not true!
> If on-line redo_file is damaged then you have
> single ability: restore your last backup.
> In all other cases database will be recovered up to the last
> committed transaction automatically!
I doubt commercial dbms's sync to disk after every transaction. They
pick a time, maybe five seconds, and see all dirty pages get flushed by
then.
What they do do is to make certain that you are restored to a consistent
state, perhaps 15 seconds ago.
--
Bruce Momjian
maillist@candle.pha.pa.us
From vadim@sable.krasnoyarsk.su Tue Nov 4 07:32:45 1997
Received: from www.krasnet.ru (www.krasnet.ru [193.125.44.86])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id HAA22066
for <maillist@candle.pha.pa.us>; Tue, 4 Nov 1997 07:32:35 -0500 (EST)
Received: from www.krasnet.ru (www.krasnet.ru [193.125.44.86]) by www.krasnet.ru (8.8.7/8.7.3) with SMTP id TAA20889; Tue, 4 Nov 1997 19:35:12 +0700 (KRS)
Sender: root@www.krasnet.ru
Message-ID: <345F1680.60E33853@sable.krasnoyarsk.su>
Date: Tue, 04 Nov 1997 19:35:12 +0700
From: "Vadim B. Mikheev" <vadim@sable.krasnoyarsk.su>
Organization: ITTS (Krasnoyarsk)
X-Mailer: Mozilla 3.01 (X11; I; FreeBSD 2.2.5-RELEASE i386)
MIME-Version: 1.0
To: Jan Wieck <wieck@sapserv.debis.de>
CC: Bruce Momjian <maillist@candle.pha.pa.us>, marc@fallon.classyad.com,
hackers@postgreSQL.org
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
References: <m0xSd44-000BFQC@orion.SAPserv.Hamburg.dsh.de>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Status: OR
wieck@sapserv.debis.de wrote:
>
> I still prefer redo logs. They should reside on a different
> disk and the possibility of loosing the database files along
> with the redo log is very small.
Agreed. This way we could don't fsync data files and
fsync both redo and pg_log. This is much faster.
Vadim
From vadim@sable.krasnoyarsk.su Tue Nov 4 08:00:58 1997
Received: from renoir.op.net (root@renoir.op.net [206.84.208.4])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id IAA22371
for <maillist@candle.pha.pa.us>; Tue, 4 Nov 1997 08:00:56 -0500 (EST)
Received: from www.krasnet.ru (www.krasnet.ru [193.125.44.86]) by renoir.op.net (o1/$ Revision: 1.14 $) with ESMTP id HAA08540 for <maillist@candle.pha.pa.us>; Tue, 4 Nov 1997 07:57:25 -0500 (EST)
Received: from www.krasnet.ru (www.krasnet.ru [193.125.44.86]) by www.krasnet.ru (8.8.7/8.7.3) with SMTP id TAA20935; Tue, 4 Nov 1997 19:59:46 +0700 (KRS)
Sender: root@www.krasnet.ru
Message-ID: <345F1C42.1F1A7590@sable.krasnoyarsk.su>
Date: Tue, 04 Nov 1997 19:59:46 +0700
From: "Vadim B. Mikheev" <vadim@sable.krasnoyarsk.su>
Organization: ITTS (Krasnoyarsk)
X-Mailer: Mozilla 3.01 (X11; I; FreeBSD 2.2.5-RELEASE i386)
MIME-Version: 1.0
To: Jan Wieck <wieck@sapserv.debis.de>
CC: Bruce Momjian <maillist@candle.pha.pa.us>, hackers@postgreSQL.org
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
References: <m0xShVQ-000BGZC@orion.SAPserv.Hamburg.dsh.de>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Status: OR
wieck@sapserv.debis.de wrote:
>
> Bruce Momjian wrote:
> > I have been thinking about re-do logs, and I think it is a good idea.
> > It would not be hard to have the queries spit out to a separate file
> > configurable by the user.
>
> This way the recovery process will be very complicated. When
> multiple backends run concurrently, there are multiple
> transactions active at the same time. And what tuples are
> affected by an update e.g. depends much on the timing.
>
> I had something different in mind. The redo log contains the
> information from the executor (e.g. the transactionId, the
> tupleId and the new tuple values when calling ExecReplace())
> and the information which transactions commit and which not.
> When recovering, those operations where the transactions
> committed are again passed to the executors functions that do
> the real updates with the values from the logfile.
It seems that this is what Oracle does, but Sybase writes queries
(with transaction ids, of 'course, and before execution) and
begin, commit/abort events <-- this is better for non-overwriting
system (shorter redo file), but, agreed, recovering is more complicated.
Vadim
From owner-pgsql-hackers@hub.org Tue Nov 4 22:35:45 1997
Received: from renoir.op.net (root@renoir.op.net [206.84.208.4])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id WAA05060
for <maillist@candle.pha.pa.us>; Tue, 4 Nov 1997 22:35:43 -0500 (EST)
Received: from hub.org (hub.org [209.47.148.200]) by renoir.op.net (o1/$ Revision: 1.14 $) with ESMTP id WAA26725 for <maillist@candle.pha.pa.us>; Tue, 4 Nov 1997 22:35:10 -0500 (EST)
Received: from localhost (majordom@localhost) by hub.org (8.8.5/8.7.5) with SMTP id WAA27875; Tue, 4 Nov 1997 22:23:14 -0500 (EST)
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Tue, 04 Nov 1997 22:20:55 -0500 (EST)
Received: (from majordom@localhost) by hub.org (8.8.5/8.7.5) id WAA24162 for pgsql-hackers-outgoing; Tue, 4 Nov 1997 22:20:50 -0500 (EST)
Received: from candle.pha.pa.us (maillist@s3-03.ppp.op.net [206.84.210.195]) by hub.org (8.8.5/8.7.5) with ESMTP id WAA22727 for <hackers@postgreSQL.org>; Tue, 4 Nov 1997 22:20:18 -0500 (EST)
Received: (from maillist@localhost)
by candle.pha.pa.us (8.8.5/8.8.5) id WAA04674;
Tue, 4 Nov 1997 22:17:52 -0500 (EST)
From: Bruce Momjian <maillist@candle.pha.pa.us>
Message-Id: <199711050317.WAA04674@candle.pha.pa.us>
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
To: vadim@sable.krasnoyarsk.su (Vadim B. Mikheev)
Date: Tue, 4 Nov 1997 22:17:52 -0500 (EST)
Cc: marc@fallon.classyad.com, hackers@postgreSQL.org
In-Reply-To: <345F14E7.28CC1042@sable.krasnoyarsk.su> from "Vadim B. Mikheev" at Nov 4, 97 07:28:23 pm
X-Mailer: ELM [version 2.4 PL25]
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Sender: owner-hackers@hub.org
Precedence: bulk
Status: OR
>
> Bruce Momjian wrote:
> >
> > >
> > > Bruce Momjian wrote:
> > > >
> > > > OK, here is a more formal description of what I am suggesting. It will
> > > > give us commercial dbms reliability with no-fsync performance.
> > > > Commercial dbms's usually only give restore up to 5 minutes before the
> > > ^^^^^^^^^^^^^^^^^^^^^^^
> > > I'm sure that this is not true!
> >
> > You may be right. This five minute figure is when you restore from your
> > previous backup, then restore from the log file.
> >
> > Can't we do something like sync every 5 seconds, rather than after every
> > transaction? It just seems like such overkill.
>
> Isn't -F and sync in crontab the same ?
OK, let me again try to marshall some (any?) support for my suggestion.
Informix version 5/7 has three levels of logging: unbuffered
logging(our normal fsync mode), buffered logging, and no logging(our no
fsync mode).
We don't have buffered logging. Buffered logging guarantees you get put
back to a consistent state after an os/server crash, usually to within
30/90 seconds. You do not have any partial transactions lying around,
but you do have some transactions that you thought were done, but are
not.
This is faster then non-buffered logging, but not as fast as no logging.
Guess what mode everyone uses? The one we don't have, buffered logging!
Unbuffered logging performance is terrible. Non-buffered logging is
used to load huge chunks of data during off-hours.
The problem we have is that we fsync every transaction, which causes a
9-times slowdown in performance on single-integer inserts.
That is a pretty heavy cost. But the alternative we give people is
no-fsync mode, where we don't sync anything, and in a crash, you could
come back with partially committed data in your database, if pg_log was
sync'ed by the database, and only some of the data pages were sync'ed,
so if any data was changing within 30 seconds of the crash, you have to
restore your previous backup.
We really need a middle solution, that gives better data integrity, for
a smaller price.
>
> >
> > Actually, I found a problem with my description. Because pg_log is not
> > fsync'ed, after a crash, pages with new transactions could have been
> > flushed to disk, but not the pg_log table that contains the transaction
> > ids. The problem is that the new backend could assign a transaction id
> > that is already in use.
>
> Impossible. Backend flushes pg_variable after fetching nex 32 xids.
My suggestion is that we don't need to flush pg_variable or pg_log that
much. My suggestion would speed up the test you do with 100 inserts
inside a single transaction vs. 100 separate inserts.
> >
> > We could set a flag upon successful shutdown, and if it is not set on
> > reboot, either do a vacuum to find the max transaction id, and
> > invalidate all them not in pg_log as synced, or increase the next
> > transaction id to some huge number and invalidate all them in between.
> >
I have a fix for the problem stated above, and it doesn't require a
vacuum.
We decide to fsync pg_variable and pg_log every 10,000 transactions or
oids. Then if the database is brought up, and it was not brought down
cleanly, you increment oid and transaction_id by 10,000, because you
know you couldn't have gotten more than that. All intermediate
transactions that are not marked committed/synced are marked aborted.
---------------------------------------------------------------------------
The problem we have with the current system is that we sync by action,
not by time interval. If you are doing tons of inserts or updates, it
is syncing after every one. What people really want is something that
will sync not after every action, but after every minute or five
minutes, so when the system is busy, the syncing every minutes is just a
small amount, and when the system is idle, no one cares if is syncs, and
no one has to wait for the sync to complete.
--
Bruce Momjian
maillist@candle.pha.pa.us
From matti@algonet.se Wed Nov 5 11:02:33 1997
Received: from smtp.algonet.se (tomei.algonet.se [194.213.74.114])
by candle.pha.pa.us (8.8.5/8.8.5) with SMTP id LAA02099
for <maillist@candle.pha.pa.us>; Wed, 5 Nov 1997 11:02:28 -0500 (EST)
Received: (qmail 6685 invoked from network); 5 Nov 1997 17:01:06 +0100
Received: from du228-6.ppp.algonet.se (HELO gamma) (root@195.100.6.228)
by tomei.algonet.se with SMTP; 5 Nov 1997 17:01:06 +0100
Sender: root
Message-ID: <34609871.27EED9D@algonet.se>
Date: Wed, 05 Nov 1997 17:02:16 +0100
From: Mattias Kregert <matti@algonet.se>
Organization: Algonet ISP
X-Mailer: Mozilla 3.0Gold (X11; I; Linux 2.0.29 i586)
MIME-Version: 1.0
To: Bruce Momjian <maillist@candle.pha.pa.us>
CC: pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
References: <199711050317.WAA04674@candle.pha.pa.us>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Status: OR
Bruce Momjian wrote:
>
> We don't have buffered logging. Buffered logging guarantees you get put
> back to a consistent state after an os/server crash, usually to within
> 30/90 seconds. You do not have any partial transactions lying around,
> but you do have some transactions that you thought were done, but are
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> not.
^^^^
>
> This is faster then non-buffered logging, but not as fast as no logging.
> Guess what mode everyone uses? The one we don't have, buffered logging!
Ouch! I would *not* like to use "buffered logging".
What's the point in having the wrong data in the database and not
knowing what updates, inserts or deletes to do to get the correct data?
That's irrecoverable loss of data. Not what *I* want. Do *you* want it?
> We really need a middle solution, that gives better data integrity, for
> a smaller price.
What I would like to have is this:
If a backend tells the frontend that a transaction has completed,
then that transaction should absolutely not get lost in case of a crash.
What is needed is a log of changes since the last backup. This
log would preferrably reside on a remote machine or at least
another disk. Then, if the power goes in the middle of a disk write,
the disk explodes and the computer goes up in flames, you can
install Postgresql on a new machine, restore the last backup and
re-run the change log.
> The problem we have with the current system is that we sync by action,
> not by time interval. If you are doing tons of inserts or updates, it
> is syncing after every one. What people really want is something that
> will sync not after every action, but after every minute or five
> minutes, so when the system is busy, the syncing every minutes is just a
> small amount, and when the system is idle, no one cares if is syncs, and
> no one has to wait for the sync to complete.
Yes, but this would only be the first step on the way to better
crash-recovery.
/* m */
From vadim@sable.krasnoyarsk.su Wed Nov 5 12:20:23 1997
Received: from renoir.op.net (root@renoir.op.net [206.84.208.4])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id MAA05156
for <maillist@candle.pha.pa.us>; Wed, 5 Nov 1997 12:20:13 -0500 (EST)
Received: from www.krasnet.ru (www.krasnet.ru [193.125.44.86]) by renoir.op.net (o1/$ Revision: 1.14 $) with ESMTP id LAA24123 for <maillist@candle.pha.pa.us>; Wed, 5 Nov 1997 11:44:49 -0500 (EST)
Received: from www.krasnet.ru (www.krasnet.ru [193.125.44.86]) by www.krasnet.ru (8.8.7/8.7.3) with SMTP id XAA23062; Wed, 5 Nov 1997 23:48:52 +0700 (KRS)
Sender: root@www.krasnet.ru
Message-ID: <3460A374.41C67EA6@sable.krasnoyarsk.su>
Date: Wed, 05 Nov 1997 23:48:52 +0700
From: "Vadim B. Mikheev" <vadim@sable.krasnoyarsk.su>
Organization: ITTS (Krasnoyarsk)
X-Mailer: Mozilla 3.01 (X11; I; FreeBSD 2.2.5-RELEASE i386)
MIME-Version: 1.0
To: Bruce Momjian <maillist@candle.pha.pa.us>
CC: marc@fallon.classyad.com, hackers@postgreSQL.org
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
References: <199711050317.WAA04674@candle.pha.pa.us>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Status: OR
Bruce Momjian wrote:
>
> OK, let me again try to marshall some (any?) support for my suggestion.
>
> Informix version 5/7 has three levels of logging: unbuffered
> logging(our normal fsync mode), buffered logging, and no logging(our no
> fsync mode).
>
> We don't have buffered logging. Buffered logging guarantees you get put
> back to a consistent state after an os/server crash, usually to within
> 30/90 seconds. You do not have any partial transactions lying around,
> but you do have some transactions that you thought were done, but are
> not.
>
> This is faster then non-buffered logging, but not as fast as no logging.
> Guess what mode everyone uses? The one we don't have, buffered logging!
>
> Unbuffered logging performance is terrible. Non-buffered logging is
> used to load huge chunks of data during off-hours.
>
> The problem we have is that we fsync every transaction, which causes a
> 9-times slowdown in performance on single-integer inserts.
>
> That is a pretty heavy cost. But the alternative we give people is
> no-fsync mode, where we don't sync anything, and in a crash, you could
> come back with partially committed data in your database, if pg_log was
> sync'ed by the database, and only some of the data pages were sync'ed,
> so if any data was changing within 30 seconds of the crash, you have to
> restore your previous backup.
>
> We really need a middle solution, that gives better data integrity, for
> a smaller price.
There is no fsync synchronization currently.
How could we be ensured that all modified data pages are flushed
when we decided to flush pg_log ?
If backend doesn't fsync data pages & pg_log at the commit time
then when he must flush them (data first) ?
This is what Oracle does:
it uses dedicated DBWR process for writing/flushing modified
data pages and LGWR process for writing/flushing redo log
(redo log is transaction log also). LGWR always flushes log pages
when committing, but durty data pages can be flushed _after_ transaction
commit when DBWR decides that it's time to do it (ala checkpoints interval).
Using redo log we could implement buffered logging quite easy.
We can even don't use dedicated processes (but flush redo before pg_log),
though having LGWR could simplify things.
Without redo log or without some fsync synchronization we can't implement
buffered logging. BTW, shared system cache could help with
fsync synchonization, but, imho, redo is better (and faster for
un-buffered logging too).
> > > Actually, I found a problem with my description. Because pg_log is not
> > > fsync'ed, after a crash, pages with new transactions could have been
> > > flushed to disk, but not the pg_log table that contains the transaction
> > > ids. The problem is that the new backend could assign a transaction id
> > > that is already in use.
> >
> > Impossible. Backend flushes pg_variable after fetching nex 32 xids.
>
> My suggestion is that we don't need to flush pg_variable or pg_log that
> much. My suggestion would speed up the test you do with 100 inserts
> inside a single transaction vs. 100 separate inserts.
>
> > >
> > > We could set a flag upon successful shutdown, and if it is not set on
> > > reboot, either do a vacuum to find the max transaction id, and
> > > invalidate all them not in pg_log as synced, or increase the next
> > > transaction id to some huge number and invalidate all them in between.
> > >
>
> I have a fix for the problem stated above, and it doesn't require a
> vacuum.
>
> We decide to fsync pg_variable and pg_log every 10,000 transactions or
> oids. Then if the database is brought up, and it was not brought down
> cleanly, you increment oid and transaction_id by 10,000, because you
> know you couldn't have gotten more than that. All intermediate
> transactions that are not marked committed/synced are marked aborted.
This is what I suppose to do by placing next available oid/xid
in shmem: this allows pre-fetch much more than 32 ids at once
without losing them when session closed.
> The problem we have with the current system is that we sync by action,
> not by time interval. If you are doing tons of inserts or updates, it
> is syncing after every one. What people really want is something that
> will sync not after every action, but after every minute or five
> minutes, so when the system is busy, the syncing every minutes is just a
> small amount, and when the system is idle, no one cares if is syncs, and
> no one has to wait for the sync to complete.
When I'm really doing tons of inserts/updates/deletes I use
BEGIN/END. But it doesn't work for multi-user environment, of 'course.
As for about what people really want, I remember that recently someone
said in user list that if one want to have 10-20 inserts/sec then he
should use mysql, but I got 25 inserts/sec on AIC-7880 & WD Enterprise
when using one session, 32 inserts/sec with two sessions inserting
in two different tables and only 20 inserts/sec with two sessions
inserting in the same table. Imho, this difference between 20 and 32
is more important thing to fix, and these results are not so bad
in comparison with others.
(BTW, we shouldn't forget about using raw devices to speed up things).
Vadim
From vadim@sable.krasnoyarsk.su Wed Nov 5 12:20:08 1997
Received: from renoir.op.net (root@renoir.op.net [206.84.208.4])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id MAA05150
for <maillist@candle.pha.pa.us>; Wed, 5 Nov 1997 12:20:07 -0500 (EST)
Received: from www.krasnet.ru (www.krasnet.ru [193.125.44.86]) by renoir.op.net (o1/$ Revision: 1.14 $) with ESMTP id LAA24889 for <maillist@candle.pha.pa.us>; Wed, 5 Nov 1997 11:59:27 -0500 (EST)
Received: from www.krasnet.ru (www.krasnet.ru [193.125.44.86]) by www.krasnet.ru (8.8.7/8.7.3) with SMTP id AAA23096; Thu, 6 Nov 1997 00:03:19 +0700 (KRS)
Sender: root@www.krasnet.ru
Message-ID: <3460A6D7.167EB0E7@sable.krasnoyarsk.su>
Date: Thu, 06 Nov 1997 00:03:19 +0700
From: "Vadim B. Mikheev" <vadim@sable.krasnoyarsk.su>
Organization: ITTS (Krasnoyarsk)
X-Mailer: Mozilla 3.01 (X11; I; FreeBSD 2.2.5-RELEASE i386)
MIME-Version: 1.0
To: Mattias Kregert <matti@algonet.se>
CC: Bruce Momjian <maillist@candle.pha.pa.us>, pgsql-hackers@postgreSQL.org
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
References: <199711050317.WAA04674@candle.pha.pa.us> <34609871.27EED9D@algonet.se>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Status: OR
Mattias Kregert wrote:
>
> Bruce Momjian wrote:
> >
> > We don't have buffered logging. Buffered logging guarantees you get put
> > back to a consistent state after an os/server crash, usually to within
> > 30/90 seconds. You do not have any partial transactions lying around,
> > but you do have some transactions that you thought were done, but are
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > not.
> ^^^^
> >
> > This is faster then non-buffered logging, but not as fast as no logging.
> > Guess what mode everyone uses? The one we don't have, buffered logging!
>
> Ouch! I would *not* like to use "buffered logging".
And I.
> What's the point in having the wrong data in the database and not
> knowing what updates, inserts or deletes to do to get the correct data?
>
> That's irrecoverable loss of data. Not what *I* want. Do *you* want it?
>
> > We really need a middle solution, that gives better data integrity, for
> > a smaller price.
>
> What I would like to have is this:
>
> If a backend tells the frontend that a transaction has completed,
> then that transaction should absolutely not get lost in case of a crash.
Agreed.
>
> What is needed is a log of changes since the last backup. This
> log would preferrably reside on a remote machine or at least
> another disk. Then, if the power goes in the middle of a disk write,
> the disk explodes and the computer goes up in flames, you can
> install Postgresql on a new machine, restore the last backup and
> re-run the change log.
Yes. And as I already said - this will speed up things because
redo flushing is faster than flushing NNN tables which can be
unflushed for some interval.
Vadim
From owner-pgsql-hackers@hub.org Wed Nov 5 12:20:39 1997
Received: from renoir.op.net (root@renoir.op.net [206.84.208.4])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id MAA05168
for <maillist@candle.pha.pa.us>; Wed, 5 Nov 1997 12:20:38 -0500 (EST)
Received: from hub.org (hub.org [209.47.148.200]) by renoir.op.net (o1/$ Revision: 1.14 $) with ESMTP id MAA25888 for <maillist@candle.pha.pa.us>; Wed, 5 Nov 1997 12:14:14 -0500 (EST)
Received: from localhost (majordom@localhost) by hub.org (8.8.5/8.7.5) with SMTP id MAA02259; Wed, 5 Nov 1997 12:02:33 -0500 (EST)
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Wed, 05 Nov 1997 12:00:21 -0500 (EST)
Received: (from majordom@localhost) by hub.org (8.8.5/8.7.5) id MAA00750 for pgsql-hackers-outgoing; Wed, 5 Nov 1997 12:00:10 -0500 (EST)
Received: from www.krasnet.ru (www.krasnet.ru [193.125.44.86]) by hub.org (8.8.5/8.7.5) with ESMTP id LAA00598 for <pgsql-hackers@postgreSQL.org>; Wed, 5 Nov 1997 11:59:45 -0500 (EST)
Received: from www.krasnet.ru (www.krasnet.ru [193.125.44.86]) by www.krasnet.ru (8.8.7/8.7.3) with SMTP id AAA23096; Thu, 6 Nov 1997 00:03:19 +0700 (KRS)
Message-ID: <3460A6D7.167EB0E7@sable.krasnoyarsk.su>
Date: Thu, 06 Nov 1997 00:03:19 +0700
From: "Vadim B. Mikheev" <vadim@sable.krasnoyarsk.su>
Organization: ITTS (Krasnoyarsk)
X-Mailer: Mozilla 3.01 (X11; I; FreeBSD 2.2.5-RELEASE i386)
MIME-Version: 1.0
To: Mattias Kregert <matti@algonet.se>
CC: Bruce Momjian <maillist@candle.pha.pa.us>, pgsql-hackers@postgreSQL.org
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
References: <199711050317.WAA04674@candle.pha.pa.us> <34609871.27EED9D@algonet.se>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Sender: owner-hackers@hub.org
Precedence: bulk
Status: OR
Mattias Kregert wrote:
>
> Bruce Momjian wrote:
> >
> > We don't have buffered logging. Buffered logging guarantees you get put
> > back to a consistent state after an os/server crash, usually to within
> > 30/90 seconds. You do not have any partial transactions lying around,
> > but you do have some transactions that you thought were done, but are
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > not.
> ^^^^
> >
> > This is faster then non-buffered logging, but not as fast as no logging.
> > Guess what mode everyone uses? The one we don't have, buffered logging!
>
> Ouch! I would *not* like to use "buffered logging".
And I.
> What's the point in having the wrong data in the database and not
> knowing what updates, inserts or deletes to do to get the correct data?
>
> That's irrecoverable loss of data. Not what *I* want. Do *you* want it?
>
> > We really need a middle solution, that gives better data integrity, for
> > a smaller price.
>
> What I would like to have is this:
>
> If a backend tells the frontend that a transaction has completed,
> then that transaction should absolutely not get lost in case of a crash.
Agreed.
>
> What is needed is a log of changes since the last backup. This
> log would preferrably reside on a remote machine or at least
> another disk. Then, if the power goes in the middle of a disk write,
> the disk explodes and the computer goes up in flames, you can
> install Postgresql on a new machine, restore the last backup and
> re-run the change log.
Yes. And as I already said - this will speed up things because
redo flushing is faster than flushing NNN tables which can be
unflushed for some interval.
Vadim
From owner-pgsql-hackers@hub.org Wed Nov 5 14:01:02 1997
Received: from renoir.op.net (root@renoir.op.net [206.84.208.4])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id OAA07017
for <maillist@candle.pha.pa.us>; Wed, 5 Nov 1997 14:00:59 -0500 (EST)
Received: from hub.org (hub.org [209.47.148.200]) by renoir.op.net (o1/$ Revision: 1.14 $) with ESMTP id NAA01759 for <maillist@candle.pha.pa.us>; Wed, 5 Nov 1997 13:52:36 -0500 (EST)
Received: from localhost (majordom@localhost) by hub.org (8.8.5/8.7.5) with SMTP id NAA03611; Wed, 5 Nov 1997 13:29:43 -0500 (EST)
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Wed, 05 Nov 1997 13:27:48 -0500 (EST)
Received: (from majordom@localhost) by hub.org (8.8.5/8.7.5) id NAA03291 for pgsql-hackers-outgoing; Wed, 5 Nov 1997 13:27:41 -0500 (EST)
Received: from candle.pha.pa.us (root@s3-03.ppp.op.net [206.84.210.195]) by hub.org (8.8.5/8.7.5) with ESMTP id NAA02823 for <hackers@postgreSQL.org>; Wed, 5 Nov 1997 13:26:20 -0500 (EST)
Received: (from maillist@localhost)
by candle.pha.pa.us (8.8.5/8.8.5) id NAA05863;
Wed, 5 Nov 1997 13:16:09 -0500 (EST)
From: Bruce Momjian <maillist@candle.pha.pa.us>
Message-Id: <199711051816.NAA05863@candle.pha.pa.us>
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
To: vadim@sable.krasnoyarsk.su (Vadim B. Mikheev)
Date: Wed, 5 Nov 1997 13:16:09 -0500 (EST)
Cc: marc@fallon.classyad.com, hackers@postgreSQL.org
In-Reply-To: <3460A374.41C67EA6@sable.krasnoyarsk.su> from "Vadim B. Mikheev" at Nov 5, 97 11:48:52 pm
X-Mailer: ELM [version 2.4 PL25]
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Sender: owner-hackers@hub.org
Precedence: bulk
Status: OR
> There is no fsync synchronization currently.
> How could we be ensured that all modified data pages are flushed
> when we decided to flush pg_log ?
> If backend doesn't fsync data pages & pg_log at the commit time
> then when he must flush them (data first) ?
My idea was to have the backend do a 'sync' that causes the OS to sync
all dirty pages, then mark all committed transactions on pg_log as
'synced'. Then sync pg_log. That way, there is a clear system where we
know everything is flushed to disk, and we mark the transactions as
synced.
The only time that synced flag is used, is when the database starts up,
and it sees that the previous shutdown was not clean.
What am I missing here?
>
> This is what Oracle does:
>
> it uses dedicated DBWR process for writing/flushing modified
> data pages and LGWR process for writing/flushing redo log
> (redo log is transaction log also). LGWR always flushes log pages
> when committing, but durty data pages can be flushed _after_ transaction
> commit when DBWR decides that it's time to do it (ala checkpoints interval).
>
> Using redo log we could implement buffered logging quite easy.
> We can even don't use dedicated processes (but flush redo before pg_log),
> though having LGWR could simplify things.
>
> Without redo log or without some fsync synchronization we can't implement
> buffered logging. BTW, shared system cache could help with
> fsync synchonization, but, imho, redo is better (and faster for
> un-buffered logging too).
>
I suggested my solution because it is clean, does flushing in one
central location(postmaster), and does quick restores.
> > > > Actually, I found a problem with my description. Because pg_log is not
> > > > fsync'ed, after a crash, pages with new transactions could have been
> > > > flushed to disk, but not the pg_log table that contains the transaction
> > > > ids. The problem is that the new backend could assign a transaction id
> > > > that is already in use.
> > >
> > > Impossible. Backend flushes pg_variable after fetching nex 32 xids.
> >
> > My suggestion is that we don't need to flush pg_variable or pg_log that
> > much. My suggestion would speed up the test you do with 100 inserts
> > inside a single transaction vs. 100 separate inserts.
> >
> > > >
> > > > We could set a flag upon successful shutdown, and if it is not set on
> > > > reboot, either do a vacuum to find the max transaction id, and
> > > > invalidate all them not in pg_log as synced, or increase the next
> > > > transaction id to some huge number and invalidate all them in between.
> > > >
> >
> > I have a fix for the problem stated above, and it doesn't require a
> > vacuum.
> >
> > We decide to fsync pg_variable and pg_log every 10,000 transactions or
> > oids. Then if the database is brought up, and it was not brought down
> > cleanly, you increment oid and transaction_id by 10,000, because you
> > know you couldn't have gotten more than that. All intermediate
> > transactions that are not marked committed/synced are marked aborted.
>
> This is what I suppose to do by placing next available oid/xid
> in shmem: this allows pre-fetch much more than 32 ids at once
> without losing them when session closed.
>
> > The problem we have with the current system is that we sync by action,
> > not by time interval. If you are doing tons of inserts or updates, it
> > is syncing after every one. What people really want is something that
> > will sync not after every action, but after every minute or five
> > minutes, so when the system is busy, the syncing every minutes is just a
> > small amount, and when the system is idle, no one cares if is syncs, and
> > no one has to wait for the sync to complete.
>
> When I'm really doing tons of inserts/updates/deletes I use
> BEGIN/END. But it doesn't work for multi-user environment, of 'course.
> As for about what people really want, I remember that recently someone
> said in user list that if one want to have 10-20 inserts/sec then he
> should use mysql, but I got 25 inserts/sec on AIC-7880 & WD Enterprise
> when using one session, 32 inserts/sec with two sessions inserting
> in two different tables and only 20 inserts/sec with two sessions
> inserting in the same table. Imho, this difference between 20 and 32
> is more important thing to fix, and these results are not so bad
> in comparison with others.
>
> (BTW, we shouldn't forget about using raw devices to speed up things).
>
> Vadim
>
--
Bruce Momjian
maillist@candle.pha.pa.us
From james@blarg.net Wed Nov 5 13:26:46 1997
Received: from animal.blarg.net (mail@animal.blarg.net [206.114.144.1])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id NAA06130
for <maillist@candle.pha.pa.us>; Wed, 5 Nov 1997 13:26:26 -0500 (EST)
Received: from animal.blarg.net (james@animal.blarg.net [206.114.144.1])
by animal.blarg.net (8.8.5/8.8.4) with SMTP
id KAA09775; Wed, 5 Nov 1997 10:26:10 -0800
Date: Wed, 5 Nov 1997 10:26:10 -0800 (PST)
From: "James A. Hillyerd" <james@blarg.net>
To: Bruce Momjian <maillist@candle.pha.pa.us>
cc: Mattias Kregert <matti@algonet.se>, pgsql-hackers@postgreSQL.org
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
In-Reply-To: <199711051615.LAA02260@candle.pha.pa.us>
Message-ID: <Pine.LNX.3.95.971105102332.6252E-100000@animal.blarg.net>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Status: OR
On Wed, 5 Nov 1997, Bruce Momjian wrote:
>
> The strange thing I am hearing is that the people who use PostgreSQL are
> more worried about data recovery from a crash than million-dollar
> companies that use commercial databases.
>
If I may throw in my 2 cents, I'd prefer to see that database in a
consistent state, with the data being up to date as of 1 minute or
less before the crash. I'd rather have higher performance than up to the
second data.
-james
[ James A. Hillyerd (JH2162) - james@blarg.net - Web Developer ]
[ http://www.blarg.net/~james/ http://www.hyperglyphics.com/ ]
[ 1024/B11C3751 CA 1C B3 A9 07 2F 57 C9 91 F4 73 F2 19 A4 C5 88 ]
From vadim@sable.krasnoyarsk.su Wed Nov 5 14:24:03 1997
Received: from renoir.op.net (root@renoir.op.net [206.84.208.4])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id OAA07830
for <maillist@candle.pha.pa.us>; Wed, 5 Nov 1997 14:24:02 -0500 (EST)
Received: from www.krasnet.ru (www.krasnet.ru [193.125.44.86]) by renoir.op.net (o1/$ Revision: 1.14 $) with ESMTP id OAA02778 for <maillist@candle.pha.pa.us>; Wed, 5 Nov 1997 14:13:45 -0500 (EST)
Received: from www.krasnet.ru (www.krasnet.ru [193.125.44.86]) by www.krasnet.ru (8.8.7/8.7.3) with SMTP id CAA23376; Thu, 6 Nov 1997 02:17:51 +0700 (KRS)
Sender: root@www.krasnet.ru
Message-ID: <3460C65E.446B9B3D@sable.krasnoyarsk.su>
Date: Thu, 06 Nov 1997 02:17:50 +0700
From: "Vadim B. Mikheev" <vadim@sable.krasnoyarsk.su>
Organization: ITTS (Krasnoyarsk)
X-Mailer: Mozilla 3.01 (X11; I; FreeBSD 2.2.5-RELEASE i386)
MIME-Version: 1.0
To: Bruce Momjian <maillist@candle.pha.pa.us>
CC: marc@fallon.classyad.com, hackers@postgreSQL.org
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
References: <199711051816.NAA05863@candle.pha.pa.us>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Status: OR
Bruce Momjian wrote:
>
> > There is no fsync synchronization currently.
> > How could we be ensured that all modified data pages are flushed
> > when we decided to flush pg_log ?
> > If backend doesn't fsync data pages & pg_log at the commit time
> > then when he must flush them (data first) ?
>
> My idea was to have the backend do a 'sync' that causes the OS to sync
> all dirty pages, then mark all committed transactions on pg_log as
> 'synced'. Then sync pg_log. That way, there is a clear system where we
> know everything is flushed to disk, and we mark the transactions as
> synced.
>
> The only time that synced flag is used, is when the database starts up,
> and it sees that the previous shutdown was not clean.
>
> What am I missing here?
Ok, I see. But we can avoid 'synced' flag: we can make (just before
sync-ing data pages) in-memory copies of "on-line" durty pg_log pages
to being written/fsynced and perform write/fsync from these copies
without stopping new commits in "on-line" page(s) (nothing must go
to disk from "on-line" log pages).
Vadim
From owner-pgsql-hackers@hub.org Wed Nov 5 14:32:25 1997
Received: from hub.org (hub.org [209.47.148.200])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id OAA08101
for <maillist@candle.pha.pa.us>; Wed, 5 Nov 1997 14:32:21 -0500 (EST)
Received: from localhost (majordom@localhost) by hub.org (8.8.5/8.7.5) with SMTP id OAA22970; Wed, 5 Nov 1997 14:26:47 -0500 (EST)
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Wed, 05 Nov 1997 14:24:59 -0500 (EST)
Received: (from majordom@localhost) by hub.org (8.8.5/8.7.5) id OAA22344 for pgsql-hackers-outgoing; Wed, 5 Nov 1997 14:24:56 -0500 (EST)
Received: from candle.pha.pa.us (maillist@s3-03.ppp.op.net [206.84.210.195]) by hub.org (8.8.5/8.7.5) with ESMTP id OAA22319 for <hackers@postgreSQL.org>; Wed, 5 Nov 1997 14:24:38 -0500 (EST)
Received: (from maillist@localhost)
by candle.pha.pa.us (8.8.5/8.8.5) id OAA07661;
Wed, 5 Nov 1997 14:22:46 -0500 (EST)
From: Bruce Momjian <maillist@candle.pha.pa.us>
Message-Id: <199711051922.OAA07661@candle.pha.pa.us>
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
To: vadim@sable.krasnoyarsk.su (Vadim B. Mikheev)
Date: Wed, 5 Nov 1997 14:22:45 -0500 (EST)
Cc: marc@fallon.classyad.com, hackers@postgreSQL.org
In-Reply-To: <3460A374.41C67EA6@sable.krasnoyarsk.su> from "Vadim B. Mikheev" at Nov 5, 97 11:48:52 pm
X-Mailer: ELM [version 2.4 PL25]
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Sender: owner-hackers@hub.org
Precedence: bulk
Status: OR
Just a clarification. When I say the postmaster issues a sync, I mean
sync(2), not fsync(2).
The sync flushes all dirty pages on all file systems. Ordinary users
can issue this, and update usually does this every 30 seconds anyway.
By using this, we let the kernel figure out which buffers are dirty. We
don't have to figure this out in the postmaster.
Then we update the pg_log table to mark those transactions as synced.
On recovery from a crash, we mark the committed transactions as
uncommitted if they do not have the synced flag.
--
Bruce Momjian
maillist@candle.pha.pa.us
From owner-pgsql-hackers@hub.org Wed Nov 5 15:11:07 1997
Received: from hub.org (hub.org [209.47.148.200])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id PAA08751
for <maillist@candle.pha.pa.us>; Wed, 5 Nov 1997 15:10:59 -0500 (EST)
Received: from localhost (majordom@localhost) by hub.org (8.8.5/8.7.5) with SMTP id PAA01986; Wed, 5 Nov 1997 15:01:24 -0500 (EST)
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Wed, 05 Nov 1997 14:59:32 -0500 (EST)
Received: (from majordom@localhost) by hub.org (8.8.5/8.7.5) id OAA01414 for pgsql-hackers-outgoing; Wed, 5 Nov 1997 14:59:28 -0500 (EST)
Received: from candle.pha.pa.us (root@s3-03.ppp.op.net [206.84.210.195]) by hub.org (8.8.5/8.7.5) with ESMTP id OAA01403 for <hackers@postgreSQL.org>; Wed, 5 Nov 1997 14:59:14 -0500 (EST)
Received: (from maillist@localhost)
by candle.pha.pa.us (8.8.5/8.8.5) id OAA08283;
Wed, 5 Nov 1997 14:53:55 -0500 (EST)
From: Bruce Momjian <maillist@candle.pha.pa.us>
Message-Id: <199711051953.OAA08283@candle.pha.pa.us>
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
To: vadim@sable.krasnoyarsk.su (Vadim B. Mikheev)
Date: Wed, 5 Nov 1997 14:53:54 -0500 (EST)
Cc: marc@fallon.classyad.com, hackers@postgreSQL.org
In-Reply-To: <3460C65E.446B9B3D@sable.krasnoyarsk.su> from "Vadim B. Mikheev" at Nov 6, 97 02:17:50 am
X-Mailer: ELM [version 2.4 PL25]
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Sender: owner-hackers@hub.org
Precedence: bulk
Status: OR
> > The only time that synced flag is used, is when the database starts up,
> > and it sees that the previous shutdown was not clean.
> >
> > What am I missing here?
>
> Ok, I see. But we can avoid 'synced' flag: we can make (just before
> sync-ing data pages) in-memory copies of "on-line" durty pg_log pages
> to being written/fsynced and perform write/fsync from these copies
> without stopping new commits in "on-line" page(s) (nothing must go
> to disk from "on-line" log pages).
[Working late tonight?]
OK, now I am lost. We need the sync'ed flag so when we start the
postmaster, and we see the database we not shut down properly, we use
the flag to clear the commit flag from comitted transactions that were
not sync'ed by the postmaster.
In my opinion, we don't need any extra copies of pg_log, we can set
those sync'ed flags while others are making changes, because before we
did our sync, we gathered a list of committed transaction ids from the
shared transaction id queue that I mentioned a while ago.
We need this queue so we can find the newly-committed transactions that
do not have a sync flag. Another way we could do this would be to scan
pg_log before we sync, getting all the committed transaction ids without
sync flags. No lock is needed on the table. If we miss some new ones,
we will get them next time we scan. The problem I saw is that there is
no way to see when to stop scanning the pg_log table for such
transactions, so I thought each backend would have to put its newly
committed transactions in a separate place. Maybe I am wrong.
This syncing method just seems so natural since we have pg_log. That is
why I keep bringing it up until people tell me I am stupid.
This transaction commit/sync stuff is complicated, and takes a while to
hash out in a group.
---------------------------------------------------------------------------
I just re-read your description, and I see what you are saying. My idea
has pg_log commit flag be real commit flags while the system is running,
but on reboot after failure, we remove the commit flags on non-synced
stuff before we start up.
Your idea is to make pg_log commit flags only appear in in-memory copies
of pg_log, and write the commit flags to disk only after the sync is
done.
Either way will work. The question is, "Which is easier?" The OS is
going to sync pg_log on its own. We would almost need a second copy of
pg_log, one copy to be used on postmaster startup, and a second to be
used by running backends, and the postmaster would make a copy of the
running backend pg_log, sync the disks, and copy it to the boot copy.
I don't see how the backend is going to figure out which pg_log pages
were modified and need to be sent to the boot copy of pg_log.
Now that I am thinking, here is a good idea. Instead of a fancy
transaction queue, what if we just have the backend record the lowest
numbered transaction they commit in a shared memory area. If the
current transaction id they commit is greater than the minimum, then
change nothing. That way, the backend could copy all pg_log pages
containing that minimum pg_log transaction id up to the most recent
pg_log page, do the sync, and copy just those to the boot copy of
pg_log.
This eliminates the transaction id queue.
The nice thing about the sync-flag in pg_log is that there is no copying
by the backend. But we would have to spin through the file to set those
sync bits. Your method just copies whole pages to the boot copy.
---------------------------------------------------------------------------
I don't want to force this idea on anyone, or annoy anyone. I just
think it needs to be considered. The concepts are unusual, so once
people get the full idea, if they don't like it, we can trash it. I
still think it holds promise.
--
Bruce Momjian
maillist@candle.pha.pa.us
From hotz@jpl.nasa.gov Wed Nov 5 15:30:18 1997
Received: from hotzsun.jpl.nasa.gov (hotzsun.jpl.nasa.gov [137.79.51.138])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id PAA09500
for <maillist@candle.pha.pa.us>; Wed, 5 Nov 1997 15:30:16 -0500 (EST)
Received: from [137.79.51.141] (hotzmac [137.79.51.141]) by hotzsun.jpl.nasa.gov (8.7.6/8.7.3) with SMTP id MAA10100; Wed, 5 Nov 1997 12:29:58 -0800 (PST)
X-Sender: hotzmail@hotzsun.jpl.nasa.gov
Message-Id: <v02140b02b0868294bbc1@[137.79.51.141]>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Date: Wed, 5 Nov 1997 12:29:58 -0800
To: Bruce Momjian <maillist@candle.pha.pa.us>,
matti@algonet.se (Mattias Kregert)
From: hotz@jpl.nasa.gov (Henry B. Hotz)
Subject: Re: [HACKERS] My $.02, was: PERFORMANCE and Good Bye, Time Travel!
Cc: pgsql-hackers@postgreSQL.org
Status: OR
At 11:15 AM 11/5/97, Bruce Momjian wrote:
>The strange thing I am hearing is that the people who use PostgreSQL are
>more worried about data recovery from a crash than million-dollar
>companies that use commercial databases.
>
>I don't get it.
I would run PG to make sure that committed transactions were really written
to disk because that seems "correct" and I don't have the kind of
performance requirements that would push me to do otherwise.
That said, I can see a need for varying performance/crash-immunity
tradeoffs, and at least *one* option in between "correct" and "unprotected"
operation would seem desirable.
Signature failed Preliminary Design Review.
Feasibility of a new signature is currently being evaluated.
h.b.hotz@jpl.nasa.gov, or hbhotz@oxy.edu
From owner-pgsql-hackers@hub.org Thu Nov 6 15:51:23 1997
Received: from hub.org (hub.org [209.47.148.200])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id PAA04634
for <maillist@candle.pha.pa.us>; Thu, 6 Nov 1997 15:51:08 -0500 (EST)
Received: from localhost (majordom@localhost) by hub.org (8.8.5/8.7.5) with SMTP id PAA24783; Thu, 6 Nov 1997 15:36:47 -0500 (EST)
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Thu, 06 Nov 1997 15:36:07 -0500 (EST)
Received: (from majordom@localhost) by hub.org (8.8.5/8.7.5) id PAA24514 for pgsql-hackers-outgoing; Thu, 6 Nov 1997 15:36:02 -0500 (EST)
Received: from guevara.bildbasen.kiruna.se (guevara.bildbasen.kiruna.se [193.45.225.110]) by hub.org (8.8.5/8.7.5) with SMTP id PAA24319 for <pgsql-hackers@postgreSQL.org>; Thu, 6 Nov 1997 15:35:32 -0500 (EST)
Received: (qmail 9764 invoked by uid 129); 6 Nov 1997 20:34:35 -0000
Date: 6 Nov 1997 20:34:35 -0000
Message-ID: <19971106203435.9763.qmail@guevara.bildbasen.kiruna.se>
From: Goran Thyni <goran@bildbasen.se>
To: pgsql-hackers@postgreSQL.org
In-reply-to: <34619E9E.622F563@algonet.se> (message from Mattias Kregert on
Thu, 06 Nov 1997 11:40:30 +0100)
Subject: [HACKERS] Re: Performance vs. Crash Recovery
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Sender: owner-hackers@hub.org
Precedence: bulk
Status: OR
I am getting quiet bored by this discussion,
if someone has a strong opinion about how this
should be done go ahead and make a test implementation
then we have something to discuss.
In the mean time, if you want best possible data protection
mount you database disk sync:ed. This is safer than any scheme
we could come up with.
D*mned slow too, so everybody should be happy. :-)
And I see no point implement a periodic sync in postmaster.
All unices has cron, why not just use that.
Or even a stupid 1-liner (ba)sh-script like:
while true; do sleep 20; sync; done
best regards,
--
---------------------------------------------
G<EFBFBD>ran Thyni, sysadm, JMS Bildbasen, Kiruna
From vadim@sable.krasnoyarsk.su Thu Nov 6 23:31:41 1997
Received: from www.krasnet.ru (www.krasnet.ru [193.125.44.86])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id XAA04723
for <maillist@candle.pha.pa.us>; Thu, 6 Nov 1997 23:31:21 -0500 (EST)
Received: from www.krasnet.ru (www.krasnet.ru [193.125.44.86]) by www.krasnet.ru (8.8.7/8.7.3) with SMTP id LAA25438; Fri, 7 Nov 1997 11:36:25 +0700 (KRS)
Sender: root@www.krasnet.ru
Message-ID: <34629AC9.15FB7483@sable.krasnoyarsk.su>
Date: Fri, 07 Nov 1997 11:36:25 +0700
From: "Vadim B. Mikheev" <vadim@sable.krasnoyarsk.su>
Organization: ITTS (Krasnoyarsk)
X-Mailer: Mozilla 3.01 (X11; I; FreeBSD 2.2.5-RELEASE i386)
MIME-Version: 1.0
To: Bruce Momjian <maillist@candle.pha.pa.us>
CC: marc@fallon.classyad.com, hackers@postgreSQL.org
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
References: <199711051953.OAA08283@candle.pha.pa.us>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Status: OR
Bruce Momjian wrote:
>
> > > The only time that synced flag is used, is when the database starts up,
> > > and it sees that the previous shutdown was not clean.
> > >
> > > What am I missing here?
> >
> > Ok, I see. But we can avoid 'synced' flag: we can make (just before
> > sync-ing data pages) in-memory copies of "on-line" durty pg_log pages
> > to being written/fsynced and perform write/fsync from these copies
> > without stopping new commits in "on-line" page(s) (nothing must go
> > to disk from "on-line" log pages).
>
> [Working late tonight?]
[Yes]
> I just re-read your description, and I see what you are saying. My idea
> has pg_log commit flag be real commit flags while the system is running,
> but on reboot after failure, we remove the commit flags on non-synced
> stuff before we start up.
>
> Your idea is to make pg_log commit flags only appear in in-memory copies
> of pg_log, and write the commit flags to disk only after the sync is
> done.
>
> Either way will work. The question is, "Which is easier?" The OS is
> going to sync pg_log on its own. We would almost need a second copy of
> pg_log, one copy to be used on postmaster startup, and a second to be
> used by running backends, and the postmaster would make a copy of the
> running backend pg_log, sync the disks, and copy it to the boot copy.
>
> I don't see how the backend is going to figure out which pg_log pages
> were modified and need to be sent to the boot copy of pg_log.
>
> Now that I am thinking, here is a good idea. Instead of a fancy
> transaction queue, what if we just have the backend record the lowest
> numbered transaction they commit in a shared memory area. If the
> current transaction id they commit is greater than the minimum, then
> change nothing. That way, the backend could copy all pg_log pages
> containing that minimum pg_log transaction id up to the most recent
> pg_log page, do the sync, and copy just those to the boot copy of
> pg_log.
>
> This eliminates the transaction id queue.
>
> The nice thing about the sync-flag in pg_log is that there is no copying
> by the backend. But we would have to spin through the file to set those
> sync bits. Your method just copies whole pages to the boot copy.
In my plans to re-design transaction system I supposed to keep in shmem
two last pg_log pages. They are most often used and using ReadBuffer/WriteBuffer
to access them is not good idea. Also, we could use spinlock instead of
lock manager to synchronize access to these pages (as I see in spin.c
spinlock-s could be shared, but only exclusive ones are used) - spinlocks
are faster.
These two last pg_log pages are "online" ones. Race condition: when one or
both of online pages becomes non-online ones, i.e. pg_log has to be expanded
when writing commit/abort of "big" xid. This is how we could handle this
in "buffered" logging (delayed fsync) mode:
When backend want to write commit/abort status he acquires exclusive
OnLineLogLock. If xid belongs to online pages then backend writes status
and releases spin. If xid is less than least xid on 1st online page then
backend releases spin and does exactly the same what he does in normal mode:
flush (write and fsync) all durty data files, lock pg_log for write, ReadBuffer,
update xid status, WriteBuffer, release write lock, flush pg_log.
If xid is greater than max xid on 2nd online page then the simplest way is
just do sync(); sync() (two times), flush 1st or both online pages,
read new page(s) into online pages space, update xid status,
release OnLineLogLock spin. We could try other ways but pg_log expanding
is rare case (32K xids in one pg_log page)...
All what postmaster will have to do is:
1. Get shared OnLineLogLock.
2. Copy 2 x 8K data to private place.
3. Release spinlock.
4. sync(); sync(); (two times!)
5. Flush online pages.
We could use -F DELAY_TIME to turn fsync delayed mode ON.
And, btw, having two bits for xact status we have only one unused
status value (0x11) currently - I would like to use this for
nested xactions and savepoints...
> I don't want to force this idea on anyone, or annoy anyone. I just
> think it needs to be considered. The concepts are unusual, so once
> people get the full idea, if they don't like it, we can trash it. I
> still think it holds promise.
Agreed.
Vadim
From owner-pgsql-hackers@hub.org Fri Nov 7 01:32:49 1997
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id BAA07651
for <maillist@candle.pha.pa.us>; Fri, 7 Nov 1997 01:32:47 -0500 (EST)
Received: from hub.org (hub.org [209.47.148.200]) by renoir.op.net (o1/$ Revision: 1.14 $) with ESMTP id XAA23328 for <maillist@candle.pha.pa.us>; Thu, 6 Nov 1997 23:46:08 -0500 (EST)
Received: from localhost (majordom@localhost) by hub.org (8.8.5/8.7.5) with SMTP id XAA19565; Thu, 6 Nov 1997 23:38:55 -0500 (EST)
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Thu, 06 Nov 1997 23:36:53 -0500 (EST)
Received: (from majordom@localhost) by hub.org (8.8.5/8.7.5) id XAA18911 for pgsql-hackers-outgoing; Thu, 6 Nov 1997 23:36:44 -0500 (EST)
Received: from www.krasnet.ru (www.krasnet.ru [193.125.44.86]) by hub.org (8.8.5/8.7.5) with ESMTP id XAA18779 for <pgsql-hackers@postgreSQL.org>; Thu, 6 Nov 1997 23:36:02 -0500 (EST)
Received: from www.krasnet.ru (www.krasnet.ru [193.125.44.86]) by www.krasnet.ru (8.8.7/8.7.3) with SMTP id LAA25448; Fri, 7 Nov 1997 11:40:29 +0700 (KRS)
Message-ID: <34629BBD.59E2B600@sable.krasnoyarsk.su>
Date: Fri, 07 Nov 1997 11:40:29 +0700
From: "Vadim B. Mikheev" <vadim@sable.krasnoyarsk.su>
Organization: ITTS (Krasnoyarsk)
X-Mailer: Mozilla 3.01 (X11; I; FreeBSD 2.2.5-RELEASE i386)
MIME-Version: 1.0
To: Bruce Momjian <maillist@candle.pha.pa.us>
CC: Mattias Kregert <matti@algonet.se>, pgsql-hackers@postgreSQL.org
Subject: Re: Sync:ing data and log (Was: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!)
References: <199711061810.NAA02118@candle.pha.pa.us>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Sender: owner-hackers@hub.org
Precedence: bulk
Status: OR
Bruce Momjian wrote:
>
> >
> > Never use sync(). Use fsync(). Other processes should take care of their
> > own syncing. If you use sync(), and you have a lot of disks, the sync
> > can
> > take half a minute if you are unlucky.
>
> We could use fsync() but then the postmaster has to know what tables
> have dirty buffers, and I don't think there is an easy way to do this.
There is one way - shared system cache...
Vadim
From vadim@sable.krasnoyarsk.su Fri Nov 7 01:31:24 1997
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id BAA07639
for <maillist@candle.pha.pa.us>; Fri, 7 Nov 1997 01:31:22 -0500 (EST)
Received: from www.krasnet.ru (www.krasnet.ru [193.125.44.86]) by renoir.op.net (o1/$ Revision: 1.14 $) with ESMTP id XAA23094 for <maillist@candle.pha.pa.us>; Thu, 6 Nov 1997 23:39:00 -0500 (EST)
Received: from www.krasnet.ru (www.krasnet.ru [193.125.44.86]) by www.krasnet.ru (8.8.7/8.7.3) with SMTP id LAA25457; Fri, 7 Nov 1997 11:43:52 +0700 (KRS)
Sender: root@www.krasnet.ru
Message-ID: <34629C87.3F54BC7E@sable.krasnoyarsk.su>
Date: Fri, 07 Nov 1997 11:43:51 +0700
From: "Vadim B. Mikheev" <vadim@sable.krasnoyarsk.su>
Organization: ITTS (Krasnoyarsk)
X-Mailer: Mozilla 3.01 (X11; I; FreeBSD 2.2.5-RELEASE i386)
MIME-Version: 1.0
To: Mattias Kregert <matti@algonet.se>
CC: Bruce Momjian <maillist@candle.pha.pa.us>, pgsql-hackers@postgreSQL.org
Subject: Re: Performance vs. Crash Recovery (Was: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!)
References: <199711051615.LAA02260@candle.pha.pa.us> <34619E9E.622F563@algonet.se>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Status: OR
Mattias Kregert wrote:
>
> > The strange thing I am hearing is that the people who use PostgreSQL are
> > more worried about data recovery from a crash than million-dollar
> > companies that use commercial databases.
> >
> > I don't get it.
>
> Perhaps the million-dollar companies have more sophisticated hardware,
> like big expensive disk arrays, big UPS:es and parallell backup
> servers?
> If so, the risk of harware failure is much smaller for them.
More of that - Informix is more stable than postgres: elog(FATAL)
occures sometime and in fsync delayed mode this will cause
of losing xaction too, not onle hard/OS failure.
Vadim
From owner-pgsql-hackers@hub.org Fri Nov 7 01:31:26 1997
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id BAA07642
for <maillist@candle.pha.pa.us>; Fri, 7 Nov 1997 01:31:24 -0500 (EST)
Received: from hub.org (hub.org [209.47.148.200]) by renoir.op.net (o1/$ Revision: 1.14 $) with ESMTP id AAA24358 for <maillist@candle.pha.pa.us>; Fri, 7 Nov 1997 00:09:47 -0500 (EST)
Received: from localhost (majordom@localhost) by hub.org (8.8.5/8.7.5) with SMTP id AAA00167; Fri, 7 Nov 1997 00:03:17 -0500 (EST)
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Fri, 07 Nov 1997 00:01:26 -0500 (EST)
Received: (from majordom@localhost) by hub.org (8.8.5/8.7.5) id AAA29427 for pgsql-hackers-outgoing; Fri, 7 Nov 1997 00:01:19 -0500 (EST)
Received: from candle.pha.pa.us (root@s3-03.ppp.op.net [206.84.210.195]) by hub.org (8.8.5/8.7.5) with ESMTP id AAA29364 for <hackers@postgreSQL.org>; Fri, 7 Nov 1997 00:01:02 -0500 (EST)
Received: (from maillist@localhost)
by candle.pha.pa.us (8.8.5/8.8.5) id XAA05565;
Thu, 6 Nov 1997 23:54:33 -0500 (EST)
From: Bruce Momjian <maillist@candle.pha.pa.us>
Message-Id: <199711070454.XAA05565@candle.pha.pa.us>
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
To: vadim@sable.krasnoyarsk.su (Vadim B. Mikheev)
Date: Thu, 6 Nov 1997 23:54:33 -0500 (EST)
Cc: marc@fallon.classyad.com, hackers@postgreSQL.org
In-Reply-To: <34629AC9.15FB7483@sable.krasnoyarsk.su> from "Vadim B. Mikheev" at Nov 7, 97 11:36:25 am
X-Mailer: ELM [version 2.4 PL25]
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Sender: owner-hackers@hub.org
Precedence: bulk
Status: OR
I was worried when you didn't respond to my last list of ideas. I
thought perhaps the idea was getting on your nerves.
I haven't dropped the idea because:
1) it offers 2-9 times speedup in database modifications
2) this is how the big commercial system handle it, and I think
we need to give users this option.
3) in the way I had it designed, it wouldn't take much work to
do it.
Anything that promises that much speedup, if it can be done easy, I say
lets consider it, even if you loose 60 seconds of changes.
> In my plans to re-design transaction system I supposed to keep in shmem
> two last pg_log pages. They are most often used and using ReadBuffer/WriteBuffer
> to access them is not good idea. Also, we could use spinlock instead of
> lock manager to synchronize access to these pages (as I see in spin.c
> spinlock-s could be shared, but only exclusive ones are used) - spinlocks
> are faster.
Ah, so you already had the idea of having on-line pages in shared memory
as part of a transaction system overhaul? Right now, does each backend
lock/read/write/unlock to get at pg_log? Wow, that is bad.
Perhaps mmap() would be a good idea. My system has msync() to flush
mmap()'ed pages to the underlying file. You would still run fsync()
after that. This may give us the best of both worlds: a shared-memory
area of variable size, and control of when it get flushed to disk. Do
other OS's have this? I have a feeling OS's with unified buffer caches
don't have this ability to determine when the underlying mmap'ed file
gets sent to the underlying file and disk.
> These two last pg_log pages are "online" ones. Race condition: when one or
> both of online pages becomes non-online ones, i.e. pg_log has to be expanded
> when writing commit/abort of "big" xid. This is how we could handle this
> in "buffered" logging (delayed fsync) mode:
>
> When backend want to write commit/abort status he acquires exclusive
> OnLineLogLock. If xid belongs to online pages then backend writes status
> and releases spin. If xid is less than least xid on 1st online page then
> backend releases spin and does exactly the same what he does in normal mode:
> flush (write and fsync) all durty data files, lock pg_log for write, ReadBuffer,
> update xid status, WriteBuffer, release write lock, flush pg_log.
> If xid is greater than max xid on 2nd online page then the simplest way is
> just do sync(); sync() (two times), flush 1st or both online pages,
> read new page(s) into online pages space, update xid status,
> release OnLineLogLock spin. We could try other ways but pg_log expanding
> is rare case (32K xids in one pg_log page)...
> All what postmaster will have to do is:
> 1. Get shared OnLineLogLock.
> 2. Copy 2 x 8K data to private place.
> 3. Release spinlock.
> 4. sync(); sync(); (two times!)
> 5. Flush online pages.
>
> We could use -F DELAY_TIME to turn fsync delayed mode ON.
>
> And, btw, having two bits for xact status we have only one unused
> status value (0x11) currently - I would like to use this for
> nested xactions and savepoints...
I saw that. By keeping two copies of pg_log, one in memory to be used
by all backend, and another that hits the disk, it certainly will work.
>
> > I don't want to force this idea on anyone, or annoy anyone. I just
> > think it needs to be considered. The concepts are unusual, so once
> > people get the full idea, if they don't like it, we can trash it. I
> > still think it holds promise.
>
> Agreed.
>
> Vadim
>
--
Bruce Momjian
maillist@candle.pha.pa.us
From owner-pgsql-hackers@hub.org Fri Nov 7 01:03:09 1997
Received: from hub.org (hub.org [209.47.148.200])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id BAA07314
for <maillist@candle.pha.pa.us>; Fri, 7 Nov 1997 01:03:05 -0500 (EST)
Received: from localhost (majordom@localhost) by hub.org (8.8.5/8.7.5) with SMTP id AAA07879; Fri, 7 Nov 1997 00:57:42 -0500 (EST)
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Fri, 07 Nov 1997 00:55:52 -0500 (EST)
Received: (from majordom@localhost) by hub.org (8.8.5/8.7.5) id AAA03918 for pgsql-hackers-outgoing; Fri, 7 Nov 1997 00:55:46 -0500 (EST)
Received: from www.krasnet.ru (www.krasnet.ru [193.125.44.86]) by hub.org (8.8.5/8.7.5) with ESMTP id AAA02961 for <hackers@postgreSQL.org>; Fri, 7 Nov 1997 00:55:18 -0500 (EST)
Received: from www.krasnet.ru (www.krasnet.ru [193.125.44.86]) by www.krasnet.ru (8.8.7/8.7.3) with SMTP id MAA25567; Fri, 7 Nov 1997 12:59:29 +0700 (KRS)
Message-ID: <3462AE40.FF6D5DF@sable.krasnoyarsk.su>
Date: Fri, 07 Nov 1997 12:59:28 +0700
From: "Vadim B. Mikheev" <vadim@sable.krasnoyarsk.su>
Organization: ITTS (Krasnoyarsk)
X-Mailer: Mozilla 3.01 (X11; I; FreeBSD 2.2.5-RELEASE i386)
MIME-Version: 1.0
To: Bruce Momjian <maillist@candle.pha.pa.us>
CC: marc@fallon.classyad.com, hackers@postgreSQL.org
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
References: <199711070454.XAA05565@candle.pha.pa.us>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Sender: owner-hackers@hub.org
Precedence: bulk
Status: OR
Bruce Momjian wrote:
>
> I was worried when you didn't respond to my last list of ideas. I
> thought perhaps the idea was getting on your nerves.
No, I was (and, unfortunately, I still) busy...
>
> I haven't dropped the idea because:
>
> 1) it offers 2-9 times speedup in database modifications
> 2) this is how the big commercial system handle it, and I think
> we need to give users this option.
> 3) in the way I had it designed, it wouldn't take much work to
> do it.
>
> Anything that promises that much speedup, if it can be done easy, I say
> lets consider it, even if you loose 60 seconds of changes.
I agreed with your un-buffered logging idea. This would be excellent
feature for un-critical dbase usings (WWW, etc).
>
> > In my plans to re-design transaction system I supposed to keep in shmem
> > two last pg_log pages. They are most often used and using ReadBuffer/WriteBuffer
> > to access them is not good idea. Also, we could use spinlock instead of
> > lock manager to synchronize access to these pages (as I see in spin.c
> > spinlock-s could be shared, but only exclusive ones are used) - spinlocks
> > are faster.
>
> Ah, so you already had the idea of having on-line pages in shared memory
> as part of a transaction system overhaul? Right now, does each backend
Yes. I hope to implement this in the next 1-2 weeks.
> lock/read/write/unlock to get at pg_log? Wow, that is bad.
Yes, he does.
>
> Perhaps mmap() would be a good idea. My system has msync() to flush
> mmap()'ed pages to the underlying file. You would still run fsync()
> after that. This may give us the best of both worlds: a shared-memory
^^^^^^^^^^^^^
> area of variable size, and control of when it get flushed to disk. Do
^^^^^^^^^^^^^^^^^^^^^
I like it. FreeBSD supports
MAP_ANON Map anonymous memory not associated with any specific file.
It would be nice to use mmap to get more "shared" memory, but I don't see
reasons to mmap any particular file to memory. Having two last pg_log pages
in memory + xact commit/abort writeback optimization (updation of commit/abort
xmin/xmax status in tuples by any scan - we already have this) reduce access
to "old" pg_log pages to zero.
> other OS's have this? I have a feeling OS's with unified buffer caches
> don't have this ability to determine when the underlying mmap'ed file
> gets sent to the underlying file and disk.
>
> > These two last pg_log pages are "online" ones. Race condition: when one or
> > both of online pages becomes non-online ones, i.e. pg_log has to be expanded
> > when writing commit/abort of "big" xid. This is how we could handle this
> > in "buffered" logging (delayed fsync) mode:
> >
> > When backend want to write commit/abort status he acquires exclusive
> > OnLineLogLock. If xid belongs to online pages then backend writes status
> > and releases spin. If xid is less than least xid on 1st online page then
> > backend releases spin and does exactly the same what he does in normal mode:
> > flush (write and fsync) all durty data files, lock pg_log for write, ReadBuffer,
> > update xid status, WriteBuffer, release write lock, flush pg_log.
> > If xid is greater than max xid on 2nd online page then the simplest way is
> > just do sync(); sync() (two times), flush 1st or both online pages,
> > read new page(s) into online pages space, update xid status,
> > release OnLineLogLock spin. We could try other ways but pg_log expanding
> > is rare case (32K xids in one pg_log page)...
> > All what postmaster will have to do is:
> > 1. Get shared OnLineLogLock.
> > 2. Copy 2 x 8K data to private place.
> > 3. Release spinlock.
> > 4. sync(); sync(); (two times!)
> > 5. Flush online pages.
> >
> > We could use -F DELAY_TIME to turn fsync delayed mode ON.
> >
> > And, btw, having two bits for xact status we have only one unused
> > status value (0x11) currently - I would like to use this for
> > nested xactions and savepoints...
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
More about this: 0x11 could mean "this _child_ transaction is committed -
you have to lookup in pg_xact_child to get parent xid and use pg_log again
to get parent xact status". If parent committed then child xact status
will be changed to 0x10 (committed) else - to 0x01 (aborted). Using this
we could get xact nesting and savepoints by starting new child xaction
inside running one...
>
> I saw that. By keeping two copies of pg_log, one in memory to be used
^^^^^^
Just two pg_log pages...
> by all backend, and another that hits the disk, it certainly will work.
Vadim
From vadim@sable.krasnoyarsk.su Fri Nov 7 01:30:59 1997
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id BAA07599
for <maillist@candle.pha.pa.us>; Fri, 7 Nov 1997 01:30:58 -0500 (EST)
Received: from www.krasnet.ru (www.krasnet.ru [193.125.44.86]) by renoir.op.net (o1/$ Revision: 1.14 $) with ESMTP id BAA26793 for <maillist@candle.pha.pa.us>; Fri, 7 Nov 1997 01:12:33 -0500 (EST)
Received: from www.krasnet.ru (www.krasnet.ru [193.125.44.86]) by www.krasnet.ru (8.8.7/8.7.3) with SMTP id NAA25592; Fri, 7 Nov 1997 13:16:39 +0700 (KRS)
Sender: root@www.krasnet.ru
Message-ID: <3462B247.ABD322C@sable.krasnoyarsk.su>
Date: Fri, 07 Nov 1997 13:16:39 +0700
From: "Vadim B. Mikheev" <vadim@sable.krasnoyarsk.su>
Organization: ITTS (Krasnoyarsk)
X-Mailer: Mozilla 3.01 (X11; I; FreeBSD 2.2.5-RELEASE i386)
MIME-Version: 1.0
To: Jan Wieck <wieck@sapserv.debis.de>
CC: Bruce Momjian <maillist@candle.pha.pa.us>, hackers@postgreSQL.org
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
References: <m0xT9nq-000BFQC@orion.SAPserv.Hamburg.dsh.de>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Status: OR
wieck@sapserv.debis.de wrote:
>
> Bruce wrote:
> >
> > > > It seems that this is what Oracle does, but Sybase writes queries
> > > > (with transaction ids, of 'course, and before execution) and
> > > > begin, commit/abort events <-- this is better for non-overwriting
> > > > system (shorter redo file), but, agreed, recovering is more complicated.
> > > >
> > > > Vadim
> > > >
> > >
> > > Writing only the queries (and only those that really modify
> > > data - no selects) would be much smarter and the redo files
> > > will be shorter. But it wouldn't fit for PostgreSQL as long
> > > as someone can submit a query like
> > >
> > > DELETE FROM xxx WHERE oid = 59337;
> >
> > Interesting point. Currently, an insert shows the OID as output in
> > psql. Perhaps we could do a little oid-manipulating to set the oid of
> > the insert.
>
> Only for simple inserts, not on
>
> INSERT INTO xxx SELECT any_type_of_merge_join;
I don't know how but Sybase handle this and IDENTITY (case of OIDs) too.
But I don't object you, Jan, just because I havn't time to do
"log queries" redo implementation and so I would like to have "log changes"
redo at least. (Actually, "log changes" is good for my production dbase
with 1 - 2 thousand updations per day).
(BTW, "incrementing" backup could be implemented without redo - I have
some thoughts about this, - but having additional recovering is good
in any case).
Vadim
From owner-pgsql-hackers@hub.org Fri Nov 7 15:42:58 1997
Received: from hub.org (hub.org [209.47.148.200])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id PAA22341
for <maillist@candle.pha.pa.us>; Fri, 7 Nov 1997 15:42:55 -0500 (EST)
Received: from localhost (majordom@localhost) by hub.org (8.8.5/8.7.5) with SMTP id PAA02769; Fri, 7 Nov 1997 15:28:54 -0500 (EST)
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Fri, 07 Nov 1997 15:24:00 -0500 (EST)
Received: (from majordom@localhost) by hub.org (8.8.5/8.7.5) id PAA01318 for pgsql-hackers-outgoing; Fri, 7 Nov 1997 15:23:52 -0500 (EST)
Received: from candle.pha.pa.us (maillist@s3-03.ppp.op.net [206.84.210.195]) by hub.org (8.8.5/8.7.5) with ESMTP id PAA00705 for <hackers@postgreSQL.org>; Fri, 7 Nov 1997 15:21:56 -0500 (EST)
Received: (from maillist@localhost)
by candle.pha.pa.us (8.8.5/8.8.5) id PAA20010;
Fri, 7 Nov 1997 15:20:10 -0500 (EST)
From: Bruce Momjian <maillist@candle.pha.pa.us>
Message-Id: <199711072020.PAA20010@candle.pha.pa.us>
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
To: vadim@sable.krasnoyarsk.su (Vadim B. Mikheev)
Date: Fri, 7 Nov 1997 15:20:10 -0500 (EST)
Cc: marc@fallon.classyad.com, hackers@postgreSQL.org
In-Reply-To: <3462AE40.FF6D5DF@sable.krasnoyarsk.su> from "Vadim B. Mikheev" at Nov 7, 97 12:59:28 pm
X-Mailer: ELM [version 2.4 PL25]
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Sender: owner-hackers@hub.org
Precedence: bulk
Status: OR
> > Anything that promises that much speedup, if it can be done easy, I say
> > lets consider it, even if you loose 60 seconds of changes.
>
> I agreed with your un-buffered logging idea. This would be excellent
> feature for un-critical dbase usings (WWW, etc).
Actually, it is buffered logging. We currently have unbuffered logging,
I think.
> > > In my plans to re-design transaction system I supposed to keep in shmem
> > > two last pg_log pages. They are most often used and using ReadBuffer/WriteBuffer
> > > to access them is not good idea. Also, we could use spinlock instead of
> > > lock manager to synchronize access to these pages (as I see in spin.c
> > > spinlock-s could be shared, but only exclusive ones are used) - spinlocks
> > > are faster.
> >
> > Ah, so you already had the idea of having on-line pages in shared memory
> > as part of a transaction system overhaul? Right now, does each backend
>
> Yes. I hope to implement this in the next 1-2 weeks.
>
> > lock/read/write/unlock to get at pg_log? Wow, that is bad.
>
> Yes, he does.
>
> >
> > Perhaps mmap() would be a good idea. My system has msync() to flush
> > mmap()'ed pages to the underlying file. You would still run fsync()
> > after that. This may give us the best of both worlds: a shared-memory
> ^^^^^^^^^^^^^
> > area of variable size, and control of when it get flushed to disk. Do
> ^^^^^^^^^^^^^^^^^^^^^
> I like it. FreeBSD supports
>
> MAP_ANON Map anonymous memory not associated with any specific file.
>
> It would be nice to use mmap to get more "shared" memory, but I don't see
> reasons to mmap any particular file to memory. Having two last pg_log pages
> in memory + xact commit/abort writeback optimization (updation of commit/abort
> xmin/xmax status in tuples by any scan - we already have this) reduce access
> to "old" pg_log pages to zero.
I totally agree. There is no advantage to mmap() vs. shared memory for
us. I thought if we could control when the mmap() gets flushed to disk,
we could let the OS handle the syncing, but I doubt this is going to be
portable.
Though, we could mmap() pg_log, and that way backends would not have to
read/write the blocks, and they could all see the same data. But with
the new scheme, they have most transaction ids in shared memory.
Interesting you mention the scan updating the transaction status. We
would have a problem here. It is possible a backend will update the
commit status of a data page, and that data page will make it to disk,
but if there is a crash before the update pg_log gets sync'ed, there
would be a partial transaction in the system.
I don't know any way that a backend would know the transaction has hit
disk, and the data commit flag could be set. You don't want to update
the commit flag of the data page until entire transaction has been
sync'ed. The only way to do that would be to have a 'commit and synced'
flag, but you want to save that for nested transactions.
Another case this could come in handy is to allow reuse of superceeded
data rows. If the transaction is committed and synced, the row space
could be reused by another transaction.
> > other OS's have this? I have a feeling OS's with unified buffer caches
> > don't have this ability to determine when the underlying mmap'ed file
> > gets sent to the underlying file and disk.
> >
> > > These two last pg_log pages are "online" ones. Race condition: when one or
> > > both of online pages becomes non-online ones, i.e. pg_log has to be expanded
> > > when writing commit/abort of "big" xid. This is how we could handle this
> > > in "buffered" logging (delayed fsync) mode:
> > >
> > > When backend want to write commit/abort status he acquires exclusive
> > > OnLineLogLock. If xid belongs to online pages then backend writes status
This confuses me. Why does a backend need to lock pg_log to update a
transaction status?
> > > and releases spin. If xid is less than least xid on 1st online page then
> > > backend releases spin and does exactly the same what he does in normal mode:
> > > flush (write and fsync) all durty data files, lock pg_log for write, ReadBuffer,
> > > update xid status, WriteBuffer, release write lock, flush pg_log.
> > > If xid is greater than max xid on 2nd online page then the simplest way is
> > > just do sync(); sync() (two times), flush 1st or both online pages,
> > > read new page(s) into online pages space, update xid status,
> > > release OnLineLogLock spin. We could try other ways but pg_log expanding
> > > is rare case (32K xids in one pg_log page)...
> > > All what postmaster will have to do is:
> > > 1. Get shared OnLineLogLock.
> > > 2. Copy 2 x 8K data to private place.
> > > 3. Release spinlock.
> > > 4. sync(); sync(); (two times!)
> > > 5. Flush online pages.
Great.
> > >
> > > We could use -F DELAY_TIME to turn fsync delayed mode ON.
> > >
> > > And, btw, having two bits for xact status we have only one unused
> > > status value (0x11) currently - I would like to use this for
> > > nested xactions and savepoints...
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> More about this: 0x11 could mean "this _child_ transaction is committed -
> you have to lookup in pg_xact_child to get parent xid and use pg_log again
> to get parent xact status". If parent committed then child xact status
> will be changed to 0x10 (committed) else - to 0x01 (aborted). Using this
> we could get xact nesting and savepoints by starting new child xaction
> inside running one...
OK.
>
> >
> > I saw that. By keeping two copies of pg_log, one in memory to be used
> ^^^^^^
> Just two pg_log pages...
Got it.
--
Bruce Momjian
maillist@candle.pha.pa.us
From owner-pgsql-hackers@hub.org Sun Nov 9 22:07:36 1997
Received: from hub.org (hub.org [209.47.148.200])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id WAA04655
for <maillist@candle.pha.pa.us>; Sun, 9 Nov 1997 22:07:30 -0500 (EST)
Received: from localhost (majordom@localhost) by hub.org (8.8.5/8.7.5) with SMTP id VAA07023; Sun, 9 Nov 1997 21:55:54 -0500 (EST)
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Sun, 09 Nov 1997 21:52:20 -0500 (EST)
Received: (from majordom@localhost) by hub.org (8.8.5/8.7.5) id VAA06174 for pgsql-hackers-outgoing; Sun, 9 Nov 1997 21:52:13 -0500 (EST)
Received: from candle.pha.pa.us (maillist@s3-03.ppp.op.net [206.84.210.195]) by hub.org (8.8.5/8.7.5) with ESMTP id VAA06092 for <hackers@postgreSQL.org>; Sun, 9 Nov 1997 21:51:58 -0500 (EST)
Received: (from maillist@localhost)
by candle.pha.pa.us (8.8.5/8.8.5) id VAA04150;
Sun, 9 Nov 1997 21:50:29 -0500 (EST)
From: Bruce Momjian <maillist@candle.pha.pa.us>
Message-Id: <199711100250.VAA04150@candle.pha.pa.us>
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel! (fwd)
To: vadim@sable.krasnoyarsk.su (Vadim B. Mikheev)
Date: Sun, 9 Nov 1997 21:50:29 -0500 (EST)
Cc: hackers@postgreSQL.org (PostgreSQL-development)
X-Mailer: ELM [version 2.4 PL25]
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Sender: owner-hackers@hub.org
Precedence: bulk
Status: OR
Forwarded message:
> > > Perhaps mmap() would be a good idea. My system has msync() to flush
> > > mmap()'ed pages to the underlying file. You would still run fsync()
> > > after that. This may give us the best of both worlds: a shared-memory
> > ^^^^^^^^^^^^^
> > > area of variable size, and control of when it get flushed to disk. Do
> > ^^^^^^^^^^^^^^^^^^^^^
> > I like it. FreeBSD supports
> >
> > MAP_ANON Map anonymous memory not associated with any specific file.
> >
> > It would be nice to use mmap to get more "shared" memory, but I don't see
> > reasons to mmap any particular file to memory. Having two last pg_log pages
> > in memory + xact commit/abort writeback optimization (updation of commit/abort
> > xmin/xmax status in tuples by any scan - we already have this) reduce access
> > to "old" pg_log pages to zero.
>
> I totally agree. There is no advantage to mmap() vs. shared memory for
> us. I thought if we could control when the mmap() gets flushed to disk,
> we could let the OS handle the syncing, but I doubt this is going to be
> portable.
>
> Though, we could mmap() pg_log, and that way backends would not have to
> read/write the blocks, and they could all see the same data. But with
> the new scheme, they have most transaction ids in shared memory.
>
> Interesting you mention the scan updating the transaction status. We
> would have a problem here. It is possible a backend will update the
> commit status of a data page, and that data page will make it to disk,
> but if there is a crash before the update pg_log gets sync'ed, there
> would be a partial transaction in the system.
>
> I don't know any way that a backend would know the transaction has hit
> disk, and the data commit flag could be set. You don't want to update
> the commit flag of the data page until entire transaction has been
> sync'ed. The only way to do that would be to have a 'commit and synced'
> flag, but you want to save that for nested transactions.
>
> Another case this could come in handy is to allow reuse of superceeded
> data rows. If the transaction is committed and synced, the row space
> could be reused by another transaction.
>
I have been thinking about the mmap() issue, and it seems a natural for
pg_log. You can have every backend mmap() pg_log. It becomes a dynamic
shared memory area that is auto-initialized to the contents of pg_log,
and all changes can be made by all backends. No locking needed. We can
also flush the changes to the underlying file. Under bsdi, you can also
have the mmap area follow you across exec() calls, so each backend
doesn't have to do anything. I want to replace exec with fork also, so
the stuff would be auto-loaded in the address space of each backend.
This way, you don't have to have two on-line pages and move them around
as pg_log grows.
The only problem remains how to mark certain transactions as synced or
force only synced transactions to hit the pg_log file itself, and data
row commit status only should be updated for synced transactions.
--
Bruce Momjian
maillist@candle.pha.pa.us
From vadim@sable.krasnoyarsk.su Sun Nov 9 23:00:58 1997
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id XAA05394
for <maillist@candle.pha.pa.us>; Sun, 9 Nov 1997 23:00:55 -0500 (EST)
Received: from www.krasnet.ru (www.krasnet.ru [193.125.44.86]) by renoir.op.net (o1/$ Revision: 1.14 $) with ESMTP id WAA25139 for <maillist@candle.pha.pa.us>; Sun, 9 Nov 1997 22:42:33 -0500 (EST)
Received: from www.krasnet.ru (www.krasnet.ru [193.125.44.86]) by www.krasnet.ru (8.8.7/8.7.3) with SMTP id KAA01845; Mon, 10 Nov 1997 10:49:25 +0700 (KRS)
Sender: root@www.krasnet.ru
Message-ID: <34668444.237C228A@sable.krasnoyarsk.su>
Date: Mon, 10 Nov 1997 10:49:24 +0700
From: "Vadim B. Mikheev" <vadim@sable.krasnoyarsk.su>
Organization: ITTS (Krasnoyarsk)
X-Mailer: Mozilla 3.01 (X11; I; FreeBSD 2.2.5-RELEASE i386)
MIME-Version: 1.0
To: Bruce Momjian <maillist@candle.pha.pa.us>
CC: marc@fallon.classyad.com, hackers@postgreSQL.org
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
References: <199711072020.PAA20010@candle.pha.pa.us>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Status: OR
Bruce Momjian wrote:
>
> > > Anything that promises that much speedup, if it can be done easy, I say
> > > lets consider it, even if you loose 60 seconds of changes.
> >
> > I agreed with your un-buffered logging idea. This would be excellent
> > feature for un-critical dbase usings (WWW, etc).
>
> Actually, it is buffered logging. We currently have unbuffered logging,
> I think.
Sorry - mistyping.
>
> Interesting you mention the scan updating the transaction status. We
> would have a problem here. It is possible a backend will update the
> commit status of a data page, and that data page will make it to disk,
> but if there is a crash before the update pg_log gets sync'ed, there
> would be a partial transaction in the system.
You're right! Currently, only system relations can be affected by this:
backend releases locks on user tables after syncing data and pg_log.
I'll keep this in mind...
> > > > These two last pg_log pages are "online" ones. Race condition: when one or
> > > > both of online pages becomes non-online ones, i.e. pg_log has to be expanded
> > > > when writing commit/abort of "big" xid. This is how we could handle this
> > > > in "buffered" logging (delayed fsync) mode:
> > > >
> > > > When backend want to write commit/abort status he acquires exclusive
> > > > OnLineLogLock. If xid belongs to online pages then backend writes status
>
> This confuses me. Why does a backend need to lock pg_log to update a
> transaction status?
What if two backends try to change xact statuses in the same byte ?
Vadim
From owner-pgsql-hackers@hub.org Sun Nov 9 23:59:50 1997
Received: from renoir.op.net (root@renoir.op.net [209.152.193.4])
by candle.pha.pa.us (8.8.5/8.8.5) with ESMTP id XAA06523
for <maillist@candle.pha.pa.us>; Sun, 9 Nov 1997 23:59:48 -0500 (EST)
Received: from hub.org (hub.org [209.47.148.200]) by renoir.op.net (o1/$ Revision: 1.14 $) with ESMTP id XAA27105 for <maillist@candle.pha.pa.us>; Sun, 9 Nov 1997 23:41:39 -0500 (EST)
Received: from localhost (majordom@localhost) by hub.org (8.8.5/8.7.5) with SMTP id XAA08860; Sun, 9 Nov 1997 23:35:42 -0500 (EST)
Received: by hub.org (TLB v0.10a (1.23 tibbs 1997/01/09 00:29:32)); Sun, 09 Nov 1997 23:31:50 -0500 (EST)
Received: (from majordom@localhost) by hub.org (8.8.5/8.7.5) id XAA07962 for pgsql-hackers-outgoing; Sun, 9 Nov 1997 23:31:43 -0500 (EST)
Received: from candle.pha.pa.us (root@s3-03.ppp.op.net [206.84.210.195]) by hub.org (8.8.5/8.7.5) with ESMTP id XAA07875 for <hackers@postgreSQL.org>; Sun, 9 Nov 1997 23:31:28 -0500 (EST)
Received: (from maillist@localhost)
by candle.pha.pa.us (8.8.5/8.8.5) id XAA05566;
Sun, 9 Nov 1997 23:17:41 -0500 (EST)
From: Bruce Momjian <maillist@candle.pha.pa.us>
Message-Id: <199711100417.XAA05566@candle.pha.pa.us>
Subject: Re: [HACKERS] PERFORMANCE and Good Bye, Time Travel!
To: vadim@sable.krasnoyarsk.su (Vadim B. Mikheev)
Date: Sun, 9 Nov 1997 23:17:41 -0500 (EST)
Cc: marc@fallon.classyad.com, hackers@postgreSQL.org
In-Reply-To: <34668444.237C228A@sable.krasnoyarsk.su> from "Vadim B. Mikheev" at Nov 10, 97 10:49:24 am
X-Mailer: ELM [version 2.4 PL25]
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Sender: owner-hackers@hub.org
Precedence: bulk
Status: OR
> > > > > These two last pg_log pages are "online" ones. Race condition: when one or
> > > > > both of online pages becomes non-online ones, i.e. pg_log has to be expanded
> > > > > when writing commit/abort of "big" xid. This is how we could handle this
> > > > > in "buffered" logging (delayed fsync) mode:
> > > > >
> > > > > When backend want to write commit/abort status he acquires exclusive
> > > > > OnLineLogLock. If xid belongs to online pages then backend writes status
> >
> > This confuses me. Why does a backend need to lock pg_log to update a
> > transaction status?
>
> What if two backends try to change xact statuses in the same byte ?
Ooo, you got me. I so hoped to prevent locking. It would be nice if:
*x |= 3;
would be atomic, but I don't think it is. Most RISC machines don't even
have an OR against a memory address, I think.
--
Bruce Momjian
maillist@candle.pha.pa.us
Reference in New Issue View Git Blame Copy Permalink