Обсуждение: BUG #12785: server process (PID 2872) was terminated by exception 0xC0000005

The following bug has been logged on the website:

Bug reference:      12785
Logged by:          Daniele
Email address:      daniele.posenato@smartec.ch
PostgreSQL version: 9.3.3
Operating system:   W7 professionale 64 bit
Description:

Hi,

For an industrial application I have the same software running on 4
different PCs with the same characteristics and configurations. Each PC has
its how postgresql database. All 4 PC are doing the same things.
On one PC sometimes (about one per week) I get the server process (PID 2872)
was terminated by exception 0xC0000005.

On the log file I have these information:
2015-02-13 22:46:05 CET LOG:  server process (PID 2872) was terminated by
exception 0xC0000005
2015-02-13 22:46:05 CET DETAIL:  Failed process was running: SELECT * FROM
highvaluelogs INNER JOIN (SELECT * FROM hvforeachpoint WHERE logID>0) AS
SQL1 ON highvaluelogs.logID=SQL1.logID ORDER BY highvaluelogs.logtimestamp
DESC;
2015-02-13 22:46:05 CET HINT:  See C include file "ntstatus.h" for a
description of the hexadecimal value.

I am not able to figure out way the query is able to crash the service. The
query is executed every 10 seconds and it is crashing the service only in
one PC.

Thanks in advance for the support.

Best regard
Daniele

Hi Daniele:

On Thu, Feb 19, 2015 at 10:55 AM, <daniele.posenato@smartec.ch> wrote:

> Operating system:   W7 professionale 64 bit
>
=E2=80=8B....

> For an industrial application I have the same software running on 4
> different PCs with the same characteristics and configurations. Each PC h=
as
> its how postgresql database. All 4 PC are doing the same things.
> On one PC sometimes (about one per week) I get the server process (PID
> 2872)
> was terminated by exception 0xC0000005.
>

=E2=80=8BIIRC =E2=80=8B
0xC0000005
=E2=80=8B is the windows code for access violation, similar to SIGSEGV, whi=
ch uses
to be caused by stray pointers and similar things. Given you have this
running in 4 machines and only got it in one it MAY be caused by faulty
hardware ( ram flips a bit, corrupts a pointer, postgres uses pointer and
gets the error ). If you can I would suggest doing a good RAM / disk / CPU
test in that machine ( postgres is one of the candidates of getting hit by
ram/disk faults because it uses a lot of them ).

=E2=80=8BBest regards
      Francisco Olarte.=E2=80=8B

Hi Daniele:

On Mon, Feb 23, 2015 at 6:09 PM, Daniele Posenato <
daniele.posenato@smartec.ch> wrote:

>  Thank you a lot for the answer, I really appreciate it. I will try to do
> what you have suggested and then I will let you know.
>

=E2=80=8BThat's ok, but I doubt I can help you more ( I abandoned Windows m=
ore than
a dozen years ago, haven't looked back, although I still remember how that
code appeared when I did something wrong in my programs ).=E2=80=8B



>
> Just for information the problem has occurred  again since  the last emai=
l
> and always on the same query.  I could understand a crash of the service =
on
> performing an update or a delete, but I have some difficulties to
> understand this on a select.  If it was an hardware problem I would expec=
t
> the service to crash also on other actions and not randomly (about once p=
er
> week) only on  a specific select (that is executed every 10  seconds).
>

=E2=80=8BIs that query consuming a lot of your resources? ( It may be due t=
o it
being lengthy or just frequent ) because in that case it makes sense.

In many applications I have 99.9% of the work / ram usage are selects, so a
random crash is normally going to hit me in one of this.

On the crashing on select stuff. Suppose you have a faulty sector or ram
location. When you write to it ( upd or del ) nothing happens, it just
sotres the bad value, when you read it ( select, part 1, reading from
disk/ram ) nothing happens, you just get bad data, say a null pointer, then
when you use ( select part 2 ) you get the fault. In fact, if a ram
location loses data written you do not notice it on writting it, or on
reading it ( unless you get a parity error ) but on using what you read
from it.

=E2=80=8BThis is a normal pattern on programming bugs too. You have an erro=
r in
some code  and store something in a random ( or not so random ) ram
location . That code seems to work ok. But then an unrelated piece of code
reads the corrupted data and crashes ( it is one of the way the buffer
overflows work, the guilty code overflows a buffer, but works, and another
chunk of code gets its data overwritten and crashes ).


>
> Is there a way to write a select that is able to crash the service?
>

=E2=80=8BWith a good database, on good hardware, with adequate ( inifinite,=
 as you
can crash any service by just joining enough copies of a table to exhaust
avalible  ) memory and disk there shouldn't be, but if you read corrupted
data or get hit by a bit flips in the middle of processing, it may Are you
able to do a full database dump ( pg_dump, not base backup ) of your
database? If you are then you are able to read all the tables, and I would
suggest trying to reindex every table if you have quiescent periods (
pg_dump does not touch indexes, so if you have good data bad corrupted
indexes that should fix it  )=E2=80=8B


>
> I will let you know the results of the hardware check after the planned
> restart.
>

=E2=80=8BI do not know ( or remember )  what your DB sizes =E2=80=8Band upt=
ime requirements
are. But I've had that kind of problems caused by corrupted disk
structures, and have being able to recover them rewritting the database,
that means dump, drop, restore, but this depends on the system, I cannot
recommend doing it, but as I said before, if I had the same aplication in 4
machines crashing randomly in only one of them I would try to triple test
the machine and dump / restore it.

Best ergards.
    Francisco Olarte.

SGkgRnJhbmNpc2NvLA0KDQpUaGFuayB5b3UgYSBsb3QgZm9yIHRoZSBhbnN3
ZXIsIEkgcmVhbGx5IGFwcHJlY2lhdGUgaXQuIEkgd2lsbCB0cnkgdG8gZG8g
d2hhdCB5b3UgaGF2ZSBzdWdnZXN0ZWQgYW5kIHRoZW4gSSB3aWxsIGxldCB5
b3Uga25vdy4NCg0KSnVzdCBmb3IgaW5mb3JtYXRpb24gdGhlIHByb2JsZW0g
aGFzIG9jY3VycmVkICBhZ2FpbiBzaW5jZSAgdGhlIGxhc3QgZW1haWwgYW5k
IGFsd2F5cyBvbiB0aGUgc2FtZSBxdWVyeS4gIEkgY291bGQgdW5kZXJzdGFu
ZCBhIGNyYXNoIG9mIHRoZSBzZXJ2aWNlIG9uIHBlcmZvcm1pbmcgYW4gdXBk
YXRlIG9yIGEgZGVsZXRlLCBidXQgSSBoYXZlIHNvbWUgZGlmZmljdWx0aWVz
IHRvIHVuZGVyc3RhbmQgdGhpcyBvbiBhIHNlbGVjdC4gIElmIGl0IHdhcyBh
biBoYXJkd2FyZSBwcm9ibGVtIEkgd291bGQgZXhwZWN0ICB0aGUgc2Vydmlj
ZSB0byBjcmFzaCBhbHNvIG9uIG90aGVyIGFjdGlvbnMgYW5kIG5vdCByYW5k
b21seSAoYWJvdXQgb25jZSBwZXIgd2Vlaykgb25seSBvbiAgYSBzcGVjaWZp
YyBzZWxlY3QgKHRoYXQgaXMgZXhlY3V0ZWQgZXZlcnkgMTAgIHNlY29uZHMp
Lg0KDQpJcyB0aGVyZSBhIHdheSB0byB3cml0ZSBhIHNlbGVjdCB0aGF0IGlz
IGFibGUgdG8gY3Jhc2ggdGhlIHNlcnZpY2U/DQoNCkkgd2lsbCBsZXQgeW91
IGtub3cgdGhlIHJlc3VsdHMgb2YgdGhlIGhhcmR3YXJlIGNoZWNrIGFmdGVy
IHRoZSBwbGFubmVkIHJlc3RhcnQuDQoNCkJlc3QgcmVnYXJkcw0KRGFuaWVs
ZQ0KDQoNCg0KRnJvbTogRnJhbmNpc2NvIE9sYXJ0ZSBbbWFpbHRvOmZvbGFy
dGVAcGVvcGxlY2FsbC5jb21dPG1haWx0bzpbbWFpbHRvOmZvbGFydGVAcGVv
cGxlY2FsbC5jb21dPg0KU2VudDogRnJpZGF5LCAyMCBGZWJydWFyeSwgMjAx
NSA1OjUxIFBNDQpUbzogRGFuaWVsZSBQb3NlbmF0bw0KQ2M6IHBnc3FsLWJ1
Z3NAcG9zdGdyZXNxbC5vcmc8bWFpbHRvOnBnc3FsLWJ1Z3NAcG9zdGdyZXNx
bC5vcmc+DQpTdWJqZWN0OiBSZTogW0JVR1NdIEJVRyAjMTI3ODU6IHNlcnZl
ciBwcm9jZXNzIChQSUQgMjg3Mikgd2FzIHRlcm1pbmF0ZWQgYnkgZXhjZXB0
aW9uIDB4QzAwMDAwMDUNCg0KSGkgRGFuaWVsZToNCg0KT24gVGh1LCBGZWIg
MTksIDIwMTUgYXQgMTA6NTUgQU0sIDxkYW5pZWxlLnBvc2VuYXRvQHNtYXJ0
ZWMuY2g8bWFpbHRvOmRhbmllbGUucG9zZW5hdG9Ac21hcnRlYy5jaD4+IHdy
b3RlOg0KT3BlcmF0aW5nIHN5c3RlbTogICBXNyBwcm9mZXNzaW9uYWxlIDY0
IGJpdA0K4oCLLi4uLg0KRm9yIGFuIGluZHVzdHJpYWwgYXBwbGljYXRpb24g
SSBoYXZlIHRoZSBzYW1lIHNvZnR3YXJlIHJ1bm5pbmcgb24gNA0KZGlmZmVy
ZW50IFBDcyB3aXRoIHRoZSBzYW1lIGNoYXJhY3RlcmlzdGljcyBhbmQgY29u
ZmlndXJhdGlvbnMuIEVhY2ggUEMgaGFzDQppdHMgaG93IHBvc3RncmVzcWwg
ZGF0YWJhc2UuIEFsbCA0IFBDIGFyZSBkb2luZyB0aGUgc2FtZSB0aGluZ3Mu
DQpPbiBvbmUgUEMgc29tZXRpbWVzIChhYm91dCBvbmUgcGVyIHdlZWspIEkg
Z2V0IHRoZSBzZXJ2ZXIgcHJvY2VzcyAoUElEIDI4NzIpDQp3YXMgdGVybWlu
YXRlZCBieSBleGNlcHRpb24gMHhDMDAwMDAwNS4NCg0K4oCLSUlSQyDigIsN
CjB4QzAwMDAwMDUNCuKAiyBpcyB0aGUgd2luZG93cyBjb2RlIGZvciBhY2Nl
c3MgdmlvbGF0aW9uLCBzaW1pbGFyIHRvIFNJR1NFR1YsIHdoaWNoIHVzZXMg
dG8gYmUgY2F1c2VkIGJ5IHN0cmF5IHBvaW50ZXJzIGFuZCBzaW1pbGFyIHRo
aW5ncy4gR2l2ZW4geW91IGhhdmUgdGhpcyBydW5uaW5nIGluIDQgbWFjaGlu
ZXMgYW5kIG9ubHkgZ290IGl0IGluIG9uZSBpdCBNQVkgYmUgY2F1c2VkIGJ5
IGZhdWx0eSBoYXJkd2FyZSAoIHJhbSBmbGlwcyBhIGJpdCwgY29ycnVwdHMg
YSBwb2ludGVyLCBwb3N0Z3JlcyB1c2VzIHBvaW50ZXIgYW5kIGdldHMgdGhl
IGVycm9yICkuIElmIHlvdSBjYW4gSSB3b3VsZCBzdWdnZXN0IGRvaW5nIGEg
Z29vZCBSQU0gLyBkaXNrIC8gQ1BVIHRlc3QgaW4gdGhhdCBtYWNoaW5lICgg
cG9zdGdyZXMgaXMgb25lIG9mIHRoZSBjYW5kaWRhdGVzIG9mIGdldHRpbmcg
aGl0IGJ5IHJhbS9kaXNrIGZhdWx0cyBiZWNhdXNlIGl0IHVzZXMgYSBsb3Qg
b2YgdGhlbSApLg0KDQrigItCZXN0IHJlZ2FyZHMNCiAgICAgIEZyYW5jaXNj
byBPbGFydGUu4oCLDQoNCgpSb2N0ZXN0IEJhcnJhY3VkYSBzY2FubmVkICE=

SGkgRnJhbmNpc2NvLA0KDQpUaGlzIGVtYWlsIGlzIGp1c3QgdG8gaW5mb3Jt
IHlvdSB0aGF0IEkgdGhpbmsgd2UgaGF2ZSBkaXNjb3ZlcmVkIHRoZSBzb3Vy
Y2Ugb2YgdGhlIGlzc3VlLiBUaGUgcHJvYmxlbSBzZWVtcyB0byBiZSByZWxh
dGVkIHRvIHRoZSBoYXJkIGRpc2ssIGV2ZW4gaWYgdGhlIHNlcnZlciB3YXMg
bmV3IHRoZSBkaXNrIHN0YXJ0ZWQgdG8gc2hvdyBzb21lIGFub21hbGllcywg
bm93IGZvciBleGFtcGxlIHdlIGhhdmUgYSBsb3Qgb2YgbG9ncyBzYXlpbmcg
dGhhdCB0aGUgZGlzayDigJxuZWVkcyB0byBiZSBjaGVja2VkIGZvciBjb25z
aXN0ZW5jeeKAnSBhbmQg4oCcdGhlIGZpbGUgc3lzdGVtIHN0cnVjdHVyZSBv
biB0aGUgZGlzayBpcyBjb3JydXB0ZWTigJ0uIEFsc28gdGhlIHBvc3RncmVz
bHEgIGNhbm5vdCAgYmUgc3RhcnRlZC4NCg0KU28gdGhhdCB5b3Ugd2VyZSBy
aWdodCwgaXQgd2FzIGFuIGhhcmR3YXJlIHByb2JsZW0uDQoNClRoYW5rcyBh
Z2FpbiBmb3IgdGhlIHN1cHBvcnQgYW5kIHRoZSB0aW1lIHlvdSBzcGVuZCBv
biBpdCwgSSBoYXZlIHJlYWxseSBhcHByZWNpYXRlZCBpdC4NCg0KQmVzdCBy
ZWdhcmRzDQpEYW5pZWxlDQoNCkZyb206IEZyYW5jaXNjbyBPbGFydGUgW21h
aWx0bzpmb2xhcnRlQHBlb3BsZWNhbGwuY29tXQ0KU2VudDogTW9uZGF5LCAy
MyBGZWJydWFyeSwgMjAxNSA2OjU2IFBNDQpUbzogRGFuaWVsZSBQb3NlbmF0
bw0KQ2M6IHBnc3FsLWJ1Z3NAcG9zdGdyZXNxbC5vcmcNClN1YmplY3Q6IFJl
OiBbQlVHU10gQlVHICMxMjc4NTogc2VydmVyIHByb2Nlc3MgKFBJRCAyODcy
KSB3YXMgdGVybWluYXRlZCBieSBleGNlcHRpb24gMHhDMDAwMDAwNQ0KDQpI
aSBEYW5pZWxlOg0KDQpPbiBNb24sIEZlYiAyMywgMjAxNSBhdCA2OjA5IFBN
LCBEYW5pZWxlIFBvc2VuYXRvIDxkYW5pZWxlLnBvc2VuYXRvQHNtYXJ0ZWMu
Y2g8bWFpbHRvOmRhbmllbGUucG9zZW5hdG9Ac21hcnRlYy5jaD4+IHdyb3Rl
Og0KVGhhbmsgeW91IGEgbG90IGZvciB0aGUgYW5zd2VyLCBJIHJlYWxseSBh
cHByZWNpYXRlIGl0LiBJIHdpbGwgdHJ5IHRvIGRvIHdoYXQgeW91IGhhdmUg
c3VnZ2VzdGVkIGFuZCB0aGVuIEkgd2lsbCBsZXQgeW91IGtub3cuDQoNCuKA
i1RoYXQncyBvaywgYnV0IEkgZG91YnQgSSBjYW4gaGVscCB5b3UgbW9yZSAo
IEkgYWJhbmRvbmVkIFdpbmRvd3MgbW9yZSB0aGFuIGEgZG96ZW4geWVhcnMg
YWdvLCBoYXZlbid0IGxvb2tlZCBiYWNrLCBhbHRob3VnaCBJIHN0aWxsIHJl
bWVtYmVyIGhvdyB0aGF0IGNvZGUgYXBwZWFyZWQgd2hlbiBJIGRpZCBzb21l
dGhpbmcgd3JvbmcgaW4gbXkgcHJvZ3JhbXMgKS7igIsNCg0KDQoNCkp1c3Qg
Zm9yIGluZm9ybWF0aW9uIHRoZSBwcm9ibGVtIGhhcyBvY2N1cnJlZCAgYWdh
aW4gc2luY2UgIHRoZSBsYXN0IGVtYWlsIGFuZCBhbHdheXMgb24gdGhlIHNh
bWUgcXVlcnkuICBJIGNvdWxkIHVuZGVyc3RhbmQgYSBjcmFzaCBvZiB0aGUg
c2VydmljZSBvbiBwZXJmb3JtaW5nIGFuIHVwZGF0ZSBvciBhIGRlbGV0ZSwg
YnV0IEkgaGF2ZSBzb21lIGRpZmZpY3VsdGllcyB0byB1bmRlcnN0YW5kIHRo
aXMgb24gYSBzZWxlY3QuICBJZiBpdCB3YXMgYW4gaGFyZHdhcmUgcHJvYmxl
bSBJIHdvdWxkIGV4cGVjdCAgdGhlIHNlcnZpY2UgdG8gY3Jhc2ggYWxzbyBv
biBvdGhlciBhY3Rpb25zIGFuZCBub3QgcmFuZG9tbHkgKGFib3V0IG9uY2Ug
cGVyIHdlZWspIG9ubHkgb24gIGEgc3BlY2lmaWMgc2VsZWN0ICh0aGF0IGlz
IGV4ZWN1dGVkIGV2ZXJ5IDEwICBzZWNvbmRzKS4NCg0K4oCLSXMgdGhhdCBx
dWVyeSBjb25zdW1pbmcgYSBsb3Qgb2YgeW91ciByZXNvdXJjZXM/ICggSXQg
bWF5IGJlIGR1ZSB0byBpdCBiZWluZyBsZW5ndGh5IG9yIGp1c3QgZnJlcXVl
bnQgKSBiZWNhdXNlIGluIHRoYXQgY2FzZSBpdCBtYWtlcyBzZW5zZS4NCklu
IG1hbnkgYXBwbGljYXRpb25zIEkgaGF2ZSA5OS45JSBvZiB0aGUgd29yayAv
IHJhbSB1c2FnZSBhcmUgc2VsZWN0cywgc28gYSByYW5kb20gY3Jhc2ggaXMg
bm9ybWFsbHkgZ29pbmcgdG8gaGl0IG1lIGluIG9uZSBvZiB0aGlzLg0KDQpP
biB0aGUgY3Jhc2hpbmcgb24gc2VsZWN0IHN0dWZmLiBTdXBwb3NlIHlvdSBo
YXZlIGEgZmF1bHR5IHNlY3RvciBvciByYW0gbG9jYXRpb24uIFdoZW4geW91
IHdyaXRlIHRvIGl0ICggdXBkIG9yIGRlbCApIG5vdGhpbmcgaGFwcGVucywg
aXQganVzdCBzb3RyZXMgdGhlIGJhZCB2YWx1ZSwgd2hlbiB5b3UgcmVhZCBp
dCAoIHNlbGVjdCwgcGFydCAxLCByZWFkaW5nIGZyb20gZGlzay9yYW0gKSBu
b3RoaW5nIGhhcHBlbnMsIHlvdSBqdXN0IGdldCBiYWQgZGF0YSwgc2F5IGEg
bnVsbCBwb2ludGVyLCB0aGVuIHdoZW4geW91IHVzZSAoIHNlbGVjdCBwYXJ0
IDIgKSB5b3UgZ2V0IHRoZSBmYXVsdC4gSW4gZmFjdCwgaWYgYSByYW0gbG9j
YXRpb24gbG9zZXMgZGF0YSB3cml0dGVuIHlvdSBkbyBub3Qgbm90aWNlIGl0
IG9uIHdyaXR0aW5nIGl0LCBvciBvbiByZWFkaW5nIGl0ICggdW5sZXNzIHlv
dSBnZXQgYSBwYXJpdHkgZXJyb3IgKSBidXQgb24gdXNpbmcgd2hhdCB5b3Ug
cmVhZCBmcm9tIGl0Lg0K4oCLVGhpcyBpcyBhIG5vcm1hbCBwYXR0ZXJuIG9u
IHByb2dyYW1taW5nIGJ1Z3MgdG9vLiBZb3UgaGF2ZSBhbiBlcnJvciBpbiBz
b21lIGNvZGUgIGFuZCBzdG9yZSBzb21ldGhpbmcgaW4gYSByYW5kb20gKCBv
ciBub3Qgc28gcmFuZG9tICkgcmFtIGxvY2F0aW9uIC4gVGhhdCBjb2RlIHNl
ZW1zIHRvIHdvcmsgb2suIEJ1dCB0aGVuIGFuIHVucmVsYXRlZCBwaWVjZSBv
ZiBjb2RlIHJlYWRzIHRoZSBjb3JydXB0ZWQgZGF0YSBhbmQgY3Jhc2hlcyAo
IGl0IGlzIG9uZSBvZiB0aGUgd2F5IHRoZSBidWZmZXIgb3ZlcmZsb3dzIHdv
cmssIHRoZSBndWlsdHkgY29kZSBvdmVyZmxvd3MgYSBidWZmZXIsIGJ1dCB3
b3JrcywgYW5kIGFub3RoZXIgY2h1bmsgb2YgY29kZSBnZXRzIGl0cyBkYXRh
IG92ZXJ3cml0dGVuIGFuZCBjcmFzaGVzICkuDQoNCg0KSXMgdGhlcmUgYSB3
YXkgdG8gd3JpdGUgYSBzZWxlY3QgdGhhdCBpcyBhYmxlIHRvIGNyYXNoIHRo
ZSBzZXJ2aWNlPw0KDQrigItXaXRoIGEgZ29vZCBkYXRhYmFzZSwgb24gZ29v
ZCBoYXJkd2FyZSwgd2l0aCBhZGVxdWF0ZSAoIGluaWZpbml0ZSwgYXMgeW91
IGNhbiBjcmFzaCBhbnkgc2VydmljZSBieSBqdXN0IGpvaW5pbmcgZW5vdWdo
IGNvcGllcyBvZiBhIHRhYmxlIHRvIGV4aGF1c3QgYXZhbGlibGUgICkgbWVt
b3J5IGFuZCBkaXNrIHRoZXJlIHNob3VsZG4ndCBiZSwgYnV0IGlmIHlvdSBy
ZWFkIGNvcnJ1cHRlZCBkYXRhIG9yIGdldCBoaXQgYnkgYSBiaXQgZmxpcHMg
aW4gdGhlIG1pZGRsZSBvZiBwcm9jZXNzaW5nLCBpdCBtYXkgQXJlIHlvdSBh
YmxlIHRvIGRvIGEgZnVsbCBkYXRhYmFzZSBkdW1wICggcGdfZHVtcCwgbm90
IGJhc2UgYmFja3VwICkgb2YgeW91ciBkYXRhYmFzZT8gSWYgeW91IGFyZSB0
aGVuIHlvdSBhcmUgYWJsZSB0byByZWFkIGFsbCB0aGUgdGFibGVzLCBhbmQg
SSB3b3VsZCBzdWdnZXN0IHRyeWluZyB0byByZWluZGV4IGV2ZXJ5IHRhYmxl
IGlmIHlvdSBoYXZlIHF1aWVzY2VudCBwZXJpb2RzICggcGdfZHVtcCBkb2Vz
IG5vdCB0b3VjaCBpbmRleGVzLCBzbyBpZiB5b3UgaGF2ZSBnb29kIGRhdGEg
YmFkIGNvcnJ1cHRlZCBpbmRleGVzIHRoYXQgc2hvdWxkIGZpeCBpdCAgKeKA
iw0KDQoNCkkgd2lsbCBsZXQgeW91IGtub3cgdGhlIHJlc3VsdHMgb2YgdGhl
IGhhcmR3YXJlIGNoZWNrIGFmdGVyIHRoZSBwbGFubmVkIHJlc3RhcnQuDQoN
CuKAi0kgZG8gbm90IGtub3cgKCBvciByZW1lbWJlciApICB3aGF0IHlvdXIg
REIgc2l6ZXMg4oCLYW5kIHVwdGltZSByZXF1aXJlbWVudHMgYXJlLiBCdXQg
SSd2ZSBoYWQgdGhhdCBraW5kIG9mIHByb2JsZW1zIGNhdXNlZCBieSBjb3Jy
dXB0ZWQgZGlzayBzdHJ1Y3R1cmVzLCBhbmQgaGF2ZSBiZWluZyBhYmxlIHRv
IHJlY292ZXIgdGhlbSByZXdyaXR0aW5nIHRoZSBkYXRhYmFzZSwgdGhhdCBt
ZWFucyBkdW1wLCBkcm9wLCByZXN0b3JlLCBidXQgdGhpcyBkZXBlbmRzIG9u
IHRoZSBzeXN0ZW0sIEkgY2Fubm90IHJlY29tbWVuZCBkb2luZyBpdCwgYnV0
IGFzIEkgc2FpZCBiZWZvcmUsIGlmIEkgaGFkIHRoZSBzYW1lIGFwbGljYXRp
b24gaW4gNCBtYWNoaW5lcyBjcmFzaGluZyByYW5kb21seSBpbiBvbmx5IG9u
ZSBvZiB0aGVtIEkgd291bGQgdHJ5IHRvIHRyaXBsZSB0ZXN0IHRoZSBtYWNo
aW5lIGFuZCBkdW1wIC8gcmVzdG9yZSBpdC4NCkJlc3QgZXJnYXJkcy4NCiAg
ICBGcmFuY2lzY28gT2xhcnRlLg0KDQoKUm9jdGVzdCBCYXJyYWN1ZGEgc2Nh
bm5lZCAhCg==