Data Corruption

[Alina ruso]

Good afternoon

Looking for some suggestions. I currently support several hundred users of an application. The application is time-tested and about as perfect as it's going to get. The majority of my time these days gets wasted fixing data corruption from viruses or other causes. The usual fix involves copying the data to another file and appending it back into a new structure or copying it out to an ASCII file and back. This works about half the time, but sometimes I get files where you might see a couple of records obviously disturbed which I delete, but then the file looks okay browsing with dBase, but will not copy out to an ASCII file and will eventually get index and other corruption error if it put back into use.

Does anyone have any good suggestions about repairing damaged corrupted files? I'm not really concerned with memo fields, just the actual files at

[Robert Bravery]

Hi,

Prevention is better than Cure.
Put the data into a backend datbase like SQL Express or FIrebird. Install
adequate AV, Monitor your servers and users

Robert

Post by Alina ruso
Good afternoon
Looking for some suggestions. I currently support several hundred users
of an application. The application is time-tested and about as perfect as
it's going to get. The majority of my time these days gets wasted fixing
data corruption from viruses or other causes. The usual fix involves
copying the data to another file and appending it back into a new
structure or copying it out to an ASCII file and back. This works about
half the time, but sometimes I get files where you might see a couple of
records obviously disturbed which I delete, but then the file looks okay
browsing with dBase, but will not copy out to an ASCII file and will
eventually get index and other corruption error if it put back into use.
Does anyone have any good suggestions about repairing damaged corrupted
files? I'm not really concerned with memo fields, just the actual files
at
---
avast! Antivirus: Inbound message clean.
Virus Database (VPS): 081003-0, 03/10/2008
Tested on: 03/10/2008 05:46:25 PM
avast! - copyright (c) 1988-2008 ALWIL Software.
http://www.avast.com

--
Web Development, Hosting, Design and Content Management Systems
http://www.integralwebsolutions.co.za



---
avast! Antivirus: Outbound message clean.
Virus Database (VPS): 081003-0, 03/10/2008
Tested on: 03/10/2008 05:48:04 PM
avast! - copyright (c) 1988-2008 ALWIL Software.
http://www.avast.com

[Mark]

For the half of the time that the writing out of the file seems to work, you might try another form of prevention by forcing a repair every x days.

That is, place a repair option on the menu that completes the repair tasks you now do. If a repair has not been done in x days (usually set to 30 unless the network they are running on is more or less stable), a message appears on any user that logs in to remind them to run a repair. The warning is on every login until someone uses the rapair option on the menu to run the repair.

This means your users are continuously repairing their own files, possibly reducing your time in that repair process.

During this user repair process, you may also check over the data to see if the repair took (i.e., programmatically identify index or other record problems). In cases where it does not work, you could display a msg to contact your office for assistance.

HTH,
Mark

Post by Alina ruso
Good afternoon
Looking for some suggestions. I currently support several hundred users of an application. The application is time-tested and about as perfect as it's going to get. The majority of my time these days gets wasted fixing data corruption from viruses or other causes. The usual fix involves copying the data to another file and appending it back into a new structure or copying it out to an ASCII file and back. This works about half the time, but sometimes I get files where you might see a couple of records obviously disturbed which I delete, but then the file looks okay browsing with dBase, but will not copy out to an ASCII file and will eventually get index and other corruption error if it put back into use.
Does anyone have any good suggestions about repairing damaged corrupted files? I'm not really concerned with memo fields, just the actual files at

[Geoff Wass [dBVIPS]]

In article <***@news-server>, ***@gmail.com
says...

Post by Alina ruso
Good afternoon
Looking for some suggestions. I currently support several hundred users of an application. The application is time-tested and about as perfect as it's going to get. The majority of my time these days gets wasted fixing data

corruption from viruses or other causes. The usual fix involves copying the data to another file and appending it back into a new structure or copying it out to an ASCII file and back. This works about half the time, but sometimes
I get files where you might see a couple of records obviously disturbed which I delete, but then the file looks okay browsing with dBase, but will not copy out to an ASCII file and will eventually get index and other corruption
error if it put back into use.

Post by Alina ruso
Does anyone have any good suggestions about repairing damaged corrupted files? I'm not really concerned with memo fields, just the actual files at

Alina,

You don't say which version of dBASE you have. If these are level 7
tables (dBASE PLUS), you could check out the dUFLP for the fixdbf7.prg.
Ken Mayer has the dUFLP at

http://www.goldenstag.net/dbase/

[Alina Ruso]

Yes lever 7 tables and I have tried that repair utility. The current file I am trying to rescue with about 100,000 records, and no dbt's. I can browse up and down the file and it looks okay at first glance, but will fail after some use. My best proof of the problem is that I can not export it to sdf or delimited. It will appear to work but only about half the records end up in the output file. That repair utility will run for hours and eventually come up with a dBASE complete crash.

Post by Geoff Wass [dBVIPS]
says...

Post by Alina ruso
Good afternoon
Looking for some suggestions. I currently support several hundred users of an application. The application is time-tested and about as perfect as it's going to get. The majority of my time these days gets wasted fixing data

corruption from viruses or other causes. The usual fix involves copying the data to another file and appending it back into a new structure or copying it out to an ASCII file and back. This works about half the time, but sometimes
I get files where you might see a couple of records obviously disturbed which I delete, but then the file looks okay browsing with dBase, but will not copy out to an ASCII file and will eventually get index and other corruption
error if it put back into use.

Post by Alina ruso
Does anyone have any good suggestions about repairing damaged corrupted files? I'm not really concerned with memo fields, just the actual files at

Alina,
You don't say which version of dBASE you have. If these are level 7
tables (dBASE PLUS), you could check out the dUFLP for the fixdbf7.prg.
Ken Mayer has the dUFLP at
http://www.goldenstag.net/dbase/
--
Geoff Wass [dBVIPS]
Montréal, Québec, Canada
.|.|.| dBASE info at http://geocities.com/geoff_wass |.|.|.
.|.|.| ---------------------------------------------------------- |.|.|.
.|.|.| IT Consultant http://Geoff_Wass.com |.|.|.

[Wesley E. Rue]

Suggestion: If as you say you can export half of the corrupted file to an
output file before dBASE crashes; why not also try to export the corrupt
file in 'reverse' order to a different output file until dBASE crashes;
then join the two output files. What's in the middle is gone.

Wes Rue

Post by Alina Ruso
Yes lever 7 tables and I have tried that repair utility. The current

file I am trying to rescue with about 100,000 records, and no dbt's. I can
browse up and down the file and it looks okay at first glance, but will
fail after some use. My best proof of the problem is that I can not export
it to sdf or delimited. It will appear to work but only about half the
records end up in the output file. That repair utility will run for hours
and eventually come up with a dBASE complete crash.

Post by Alina Ruso

Post by Geoff Wass [dBVIPS]
says...

Post by Alina ruso
Good afternoon
Looking for some suggestions. I currently support several hundred

users of an application. The application is time-tested and about as
perfect as it's going to get. The majority of my time these days gets wasted
fixing data

Post by Alina Ruso

Post by Geoff Wass [dBVIPS]
corruption from viruses or other causes. The usual fix involves copying

the data to another file and appending it back into a new structure or
copying it out to an ASCII file and back. This works about half the time,
but sometimes

Post by Alina Ruso

Post by Geoff Wass [dBVIPS]
I get files where you might see a couple of records obviously disturbed

which I delete, but then the file looks okay browsing with dBase, but will
not copy out to an ASCII file and will eventually get index and other
corruption

Post by Alina Ruso

Post by Geoff Wass [dBVIPS]
error if it put back into use.

Post by Alina ruso
Does anyone have any good suggestions about repairing damaged

corrupted files? I'm not really concerned with memo fields, just the actual
files at

Post by Alina Ruso

Post by Geoff Wass [dBVIPS]
Alina,
You don't say which version of dBASE you have. If these are level 7
tables (dBASE PLUS), you could check out the dUFLP for the fixdbf7.prg.
Ken Mayer has the dUFLP at
http://www.goldenstag.net/dbase/
--
Geoff Wass [dBVIPS]
Montréal, Québec, Canada
.|.|.| dBASE info at http://geocities.com/geoff_wass |.|.|.
.|.|.| ---------------------------------------------------------- |.|.|.
.|.|.| IT Consultant http://Geoff_Wass.com |.|.|.

[Wesley E. Rue]

Suggestion: If as you say you can export half of the corrupted file to an
output file before dBASE crashes;
why not also try to export the corrupt file in 'reverse' order to a
different output file until dBASE crashes;
then join the two output files. What's in the middle is gone. (Have no
idea if this can be done or will work.)

Wes Rue

Post by Alina Ruso
Yes lever 7 tables and I have tried that repair utility. The current

file I am trying to rescue with about 100,000 records, and no dbt's. I can
browse up and down the file and it looks okay at first glance, but will
fail after some use. My best proof of the problem is that I can not export
it to sdf or delimited. It will appear to work but only about half the
records end up in the output file. That repair utility will run for hours
and eventually come up with a dBASE complete crash.

Post by Alina Ruso

Post by Geoff Wass [dBVIPS]
says...

Post by Alina ruso
Good afternoon
Looking for some suggestions. I currently support several hundred

users of an application. The application is time-tested and about as
perfect as it's going to get. The majority of my time these days gets wasted
fixing data

Post by Alina Ruso

Post by Geoff Wass [dBVIPS]
corruption from viruses or other causes. The usual fix involves copying

the data to another file and appending it back into a new structure or
copying it out to an ASCII file and back. This works about half the time,
but sometimes

Post by Alina Ruso

Post by Geoff Wass [dBVIPS]
I get files where you might see a couple of records obviously disturbed

which I delete, but then the file looks okay browsing with dBase, but will
not copy out to an ASCII file and will eventually get index and other
corruption

Post by Alina Ruso

Post by Geoff Wass [dBVIPS]
error if it put back into use.

Post by Alina ruso
Does anyone have any good suggestions about repairing damaged

corrupted files? I'm not really concerned with memo fields, just the actual
files at

Post by Alina Ruso

Post by Geoff Wass [dBVIPS]
Alina,
You don't say which version of dBASE you have. If these are level 7
tables (dBASE PLUS), you could check out the dUFLP for the fixdbf7.prg.
Ken Mayer has the dUFLP at
http://www.goldenstag.net/dbase/
--
Geoff Wass [dBVIPS]
Montréal, Québec, Canada
.|.|.| dBASE info at http://geocities.com/geoff_wass |.|.|.
.|.|.| ---------------------------------------------------------- |.|.|.
.|.|.| IT Consultant http://Geoff_Wass.com |.|.|.

[Geoff Wass [dBVIPS]]

In article <B3C#D#***@news-server>, ***@gmail.com
says...

Post by Alina Ruso
Yes lever 7 tables and I have tried that repair utility. The current file I am trying to rescue with about 100,000 records, and no dbt's. I can browse up and down the file and it looks okay at first glance, but will fail

after some use. My best proof of the problem is that I can not export it to sdf or delimited. It will appear to work but only about half the records end up in the output file. That repair utility will run for hours and eventually
come up with a dBASE complete crash.


Alina,

It sounds like you have at least one bad character in one or more rows
of your table. Look at the rows where the export first fails and see if
there is unprintable characters or a chr(0) which dBASE will treat as a
end-of-file (EOF) marker.

[Alina Ruso]

I seem to have corrupt data someplace in the file but cannot see it visually. I cannot export the file to an ASCII file. Maybe there are some invisible ASCII characters in the file which are affecting the operation.

Can someone suggest a test for identifying bad ASCII characters in a field?

I have run into this problem before and likely will again in the future so I'm looking to understand the problem as best as possible

Alina

Post by Geoff Wass [dBVIPS]
says...

Post by Alina Ruso
Yes lever 7 tables and I have tried that repair utility. The current file I am trying to rescue with about 100,000 records, and no dbt's. I can browse up and down the file and it looks okay at first glance, but will fail

after some use. My best proof of the problem is that I can not export it to sdf or delimited. It will appear to work but only about half the records end up in the output file. That repair utility will run for hours and eventually
come up with a dBASE complete crash.
Alina,
It sounds like you have at least one bad character in one or more rows
of your table. Look at the rows where the export first fails and see if
there is unprintable characters or a chr(0) which dBASE will treat as a
end-of-file (EOF) marker.
--
Geoff Wass [dBVIPS]
Montréal, Québec, Canada
.|.|.| dBASE info at http://geocities.com/geoff_wass |.|.|.
.|.|.| ---------------------------------------------------------- |.|.|.
.|.|.| IT Consultant http://Geoff_Wass.com |.|.|.

[Mervyn Bick]

Post by Alina Ruso
I seem to have corrupt data someplace in the file but cannot see it
visually. I cannot export the file to an ASCII file. Maybe there are
some invisible ASCII characters in the file which are affecting the
operation.
Can someone suggest a test for identifying bad ASCII characters in a field?
I have run into this problem before and likely will again in the future
so I'm looking to understand the problem as best as possible

About 2 weeks ago, when you first raised this question, I posted some
example code which would extract the data from a .dbf file and save it as
a text file in SDF format. At the same time any non-printing characters
would be converted to spaces. The program reads the .dbf file but makes
no changes to it so there is no chance of doing any further damage to your
file.

You would obviously need to change the file names I used in the example
but did you try this program on your corrupt file? Did it work or were
there problems?

Mervyn.

[Alina Ruso]

I searched for your email and could not find your post with that code.
thank you in advance
Alina

Post by Mervyn Bick

Post by Alina Ruso
I seem to have corrupt data someplace in the file but cannot see it
visually. I cannot export the file to an ASCII file. Maybe there are
some invisible ASCII characters in the file which are affecting the
operation.
Can someone suggest a test for identifying bad ASCII characters in a field?
I have run into this problem before and likely will again in the future
so I'm looking to understand the problem as best as possible

About 2 weeks ago, when you first raised this question, I posted some
example code which would extract the data from a .dbf file and save it as
a text file in SDF format. At the same time any non-printing characters
would be converted to spaces. The program reads the .dbf file but makes
no changes to it so there is no chance of doing any further damage to your
file.
You would obviously need to change the file names I used in the example
but did you try this program on your corrupt file? Did it work or were
there problems?
Mervyn.

[Mervyn Bick]

Post by Alina Ruso
I searched for your email and could not find your post with that code.
thank you in advance
Alina

Save all of the code below my signature in a .prg file. Run it just as it
is to see that it works with a "clean" file. I have used library.dbf
simply because it's there and it is a reasonable size. You may need to
change path names to suit your machine. The file it produces is slightly
different to the file dBase would produce if you exported the data in SDF
format. This is because library.dbf has a memo field. dBase would ignore
the memo field and my code writes out the block number where the memo is
stored in the .dbt file. Don't worry about it. If you have a memo field
in your file we'll deal with it later. The impotant thing is to try and
save your data.

Change the IN_FILE and OUT_FILE names to suit. If your .dbf file uses
DBASELOCK change the false to true. If you're not sure and can't open the
.dbf file to check, run the program twice using different OUT_FILE names.

If DBASELOCK is incorrectly set true then 16 characters at the end of each
line will be lost. If DBASELOCK is incorrectly set false then an extra 16
characters, probably spaces but there may be debris from previous locks,
will be added to each line.

Once you have a text file you will have to inspect the file to see if the
data is OK. The file will probably be too big for Notepad which I believe
tries to take the entire file into memory but Wordpad or Word, if you have
it, should do the job. If you have a problem it is simple enough to
create something to step through the text file. If all else fails you
could TYPE <filename> | MORE at a DOS window command line. Mind you,
you'd have to be a masochist to tackle a big file this way. <g>

If you succeed in saving the data you will be able to recreate the table
structure and import in SDF format. Date fields and SDF format are not
directly compatible so if you have date fields (or memo fields) a bit of
fancy footwork will be needed but, again, we'll deal with that later if we
need to.



Mervyn.

******** Start of example code *******

#define IN_FILE "c:\duflp\library.dbf"
#define OUT_FILE "c:\duflp\examples\libsdf.txt"
#define lIsDBASELOCK false

clear
fIn = new File()
fOut = new File()
fIn.open(IN_FILE)
fOut.create(OUT_FILE)
fIn.seek(4)
nRead1 = asc(fIn.read(1))
nread2 = asc(fin.read(1))
nRead3 = asc(fIn.read(1))
nread4 = asc(fin.read(1))
nNO_RECS = nRead4*256*256*256+nRead3*256*256+nRead2*256+nRead1
nRead1 = asc(fIn.read(1))
nread2 = asc(fin.read(1))
nRead3 = asc(fIn.read(1))
nread4 = asc(fin.read(1))
nSTART = nRead2*256+nRead1
nREC_LENGTH = nRead4*256+nRead3
if lIsDBASELOCK
nBlockLen = nREC_LENGTH - 17
else
nBlockLen = nREC_LENGTH - 1
endif
nCount = 0
fIn.seek(nSTART )
do while nCount =< nNO_RECS
nCount += 1
if nCount%100=0
? nCount
endif
cDelete = fIn.read(1)
if cDelete = "*"
fIn.seek(nREC_LENGTH-1,1)
else
cStr = ""
for n = 1 to nBlockLen
cReadChar = fIn.read(1)
if asc(cReadChar) =< 32
cReadChar = " "
endif
cStr += cReadChar
next
if lIsDBASELOCK
fIN.seek(16,1)
endif
fOut.puts(cStr)
endif
enddo
fIn.close()
fOut.close()

********* End of example code *******