Windows Server - Files deleted but free space doesn't increase

Asked By JaimeZ on 06-Oct-08 11:56 AM
This is a follow-up to a previous post where I am organizing several TB of
files. The proprietary software I'm using only copies files, it doesn't move
them.

So after confirming the successful copy of the database files, I go into the
old directory to delete them. Windows (Server 2003) tells me "the files you
are trying to delete are too big for the Recycle Bin. Would you like to
permanently delete them?"
I click "Yes to all."

The files and their directories disappear but no space is freed up on the
drive. I'll bet I've deleted over 100GB so far but the free space remains
unchanged. I've tried rebooting to no avail. Now I'm running checkdisk on the
drive, but any other suggestions y'all might have would be helpful.

I do not have Norton installed on this machine. It's just a straight-up
Server 2003 install.

Thanks in advance!




Pegasus \(MVP\) replied on 06-Oct-08 02:25 PM
What happens when you delete some of those big files from the Command
Prompt?
JaimeZ replied on 08-Oct-08 09:25 AM
Sorry for the delay, I had to wait for chkdisk to finish and then last-minute
trip out of town.

Deleting from command prompt does the same thing. 58GB of files appear to be
gone but no space is freed up. Still only 1GB free on the drive. Now I've
deleted close to 200GB and still only 1GB "free." This is bizarre.

Thanks again for your thoughts!
Pegasus \(MVP\) replied on 08-Oct-08 12:19 PM
This is getting a little bizarre. Let's put your machine on the lie detector
by running a set of non-GUI commands. Copy the lines below into
c:\test.bat, then run c:\test.bat from a Command Prompt:
@echo off
set size=100000000
if exist c:\big.bin del c:\big.bin

echo %date% %time% > c:\test.txt
echo.
echo Before creating the big test file >> c:\test.txt
fsutil volume diskfree c: >> c:\test.txt

fsutil file createnew c:\big.bin %size%
echo. >> c:\test.txt
echo After creating the big test file >> c:\test.txt
fsutil volume diskfree c: >> c:\test.txt

del c:\big.bin
echo. >> c:\test.txt
echo After deleting the big test file >> c:\test.txt
fsutil volume diskfree c: >> c:\test.txt

echo Pausing for a minute . . .
ping localhost -n 60 > nul
echo. >> c:\test.txt
echo One minute later . . . >> c:\test.txt
fsutil volume diskfree c: >> c:\test.txt
notepad c:\test.txt

The batch file will create and delete a 100 MByte file while monitoring the
amount of free disk space. Post the contents of the notepad screen you see
at the end.
JaimeZ replied on 08-Oct-08 01:37 PM
Well that seems to have run okay.
--------------------------------
Wed 10/08/2008 12:30:15.73
Before creating the big test file
Total # of free bytes        : 1145970688
Total # of bytes             : 5247106805760
Total # of avail free bytes  : 1145970688

After creating the big test file
Total # of free bytes        : 1045966848
Total # of bytes             : 5247106805760
Total # of avail free bytes  : 1045966848

After deleting the big test file
Total # of free bytes        : 1145970688
Total # of bytes             : 5247106805760
Total # of avail free bytes  : 1145970688

One minute later . . .
Total # of free bytes        : 1145970688
Total # of bytes             : 5247106805760
Total # of avail free bytes  : 1145970688
--------------------------------

So why would that work and clear up space when
del f:\1\*.*
(28GB in 100,000 files) does nothing?

Jim
Pegasus \(MVP\) replied on 08-Oct-08 03:05 PM
The test you ran shows clearly that the amount of free space decreased when
you created a large file and it increased when you deleted this same file.
Conclusions:
- File deletion does work as expected.
- When you think you're deleting one of your own big files
then you're deleting something other than that file.

I suspect you deleted a link. The file behind the link is still there.
JaimeZ replied on 08-Oct-08 04:56 PM
I can certainly understand why you would say that, but here's what confuses me:
Let's say I want to delete f:\1\*.*, which as I said is 100,000 files for
28GB. I determine this two ways: first, in the window status bar it says
for "1," it says "100,000 files, 28GB."

So if I right-click -> delete the directory numbered 1, OR if I go into the
command prompt to do that, the directory goes away. But as I say, the free
space doesn't increase. I've been doing some further research and have read
that UNIX occasionally stores files in more than one location, but I haven't
seen anything along those lines for NTFS. How else would you go about
deleting the files?

(I guess what I'm saying is that I'm 99% sure I am not deleting a shortcut.)

Thanks for all of your help, I *do* appreciate your thoughts!

Jim
JaimeZ replied on 08-Oct-08 05:37 PM
Additionally, I can select, say, "50MB" worth of files and delete them and
the free bytes don't change a bit. If they were links I'd expect the free
space to increase *a little*, right? This is confusing to me. Does it maybe
have something to do with the RAID5 trying to keep data around?
Pegasus \(MVP\) replied on 08-Oct-08 05:43 PM
The strange thing is that when I give you a self-logging batch file to
create and delete a big file then things work out exactly as expected. On
the other hand, when you do things manually then they don't. It seems
necessary to repeat the deletion exercise, but this time with ***your***
files and with the process documenting itself. Try this batch file:

@echo off
set Active=no
set Drive=F:
set Folder=%Drive%\Some Folder Name

echo %date% %time% > c:\test.txt
echo.
echo Before emptying "%folder%" >> c:\test.txt
dir "%folder%" >> c:\test.txt
fsutil volume diskfree %Drive% >> c:\test.txt

if /i %Active%==yes (
del  /q "%folder%\*.*"  1>>c:\test.txt 2>>&1
) else (
echo del /q "%folder%\*.*"
)
echo. >> c:\test.txt
echo After emptying "%folder%" >> c:\test.txt
dir "%folder%" >> c:\test.txt
fsutil volume diskfree %Drive% >> c:\test.txt

notepad c:\test.txt

Adjust lines 3 and 4 to suit your environment. Do not add any double quotes
to the folder name and leave the "Set active=no" line as it is. Now run the
batch file from the Command Prompt. It will show you the "delete" command
used if it was active. If you're happy that the command is correct, modify
Line 2 like so:
set Active=yes
Do not add any spaces other than between "set" and "Active". Now run the
batch file and examine the evidence.
JaimeZ replied on 08-Oct-08 06:15 PM
Prepare for further baffling:
-------------------------------------------------

Wed 10/08/2008 17:04:02.14
Before emptying "F:\PFPS\NGA\3\rpf\cib1\X"
Volume in drive F is GPLDisk
Volume Serial Number is 30A9-7549

Directory of F:\PFPS\NGA\3\rpf\cib1\X

10/08/2008  04:32 PM    <DIR>          .
10/08/2008  04:32 PM    <DIR>          ..
05/22/2006  06:01 AM           290,080 01CYEE2B.I42
05/22/2006  06:01 AM           290,080 01CYEF2B.I42
05/22/2006  06:01 AM           290,080 01CYEG2B.I42
05/22/2006  06:01 AM           290,080 01CYEH2B.I42
(snip)
10/21/2005  08:48 AM           290,823 021GXN1B.I42
10/21/2005  08:48 AM           290,823 021GXP1B.I42
93830 File(s) 27,216,271,166 bytes
2 Dir(s)   1,140,289,536 bytes free
Total # of free bytes        : 1140293632
Total # of bytes             : 5247106805760
Total # of avail free bytes  : 1140293632

After emptying "F:\PFPS\NGA\3\rpf\cib1\X"
Volume in drive F is GPLDisk
Volume Serial Number is 30A9-7549

Directory of F:\PFPS\NGA\3\rpf\cib1\X

10/08/2008  05:05 PM    <DIR>          .
10/08/2008  05:05 PM    <DIR>          ..
0 File(s)              0 bytes
2 Dir(s)   1,164,267,520 bytes free
Total # of free bytes        : 1164267520
Total # of bytes             : 5247106805760
Total # of avail free bytes  : 1164267520

--------------------------------------------------------------

How the heck does that work? According to calc.exe I should now have
28,356,564,798 free bytes.


Jim
Pegasus \(MVP\) replied on 08-Oct-08 06:53 PM
All of this can be explained if the two following conditions are met:
- Drive F: is an NTFS volume with file compression enabled.
- Your files are highly compressible.

You can test their compressibility by zipping up one of them and comparing
its uncompressed with its zipped-up size.

You could confirm my suspicion by running this command:
dir f:\ /s | find /i "bytes"
It will probably show a final figure that exceeds your partition capacity.
JaimeZ replied on 08-Oct-08 07:54 PM
That's a clever answer!

But file compression is not enabled. (At least those directories aren't
blue.) Also, without going into too much detail, I believe those .I42 files
are some sort of bitmap. Definately not text.

Also, even if that was the case, if I had 1TB free before I began the
move-via-copy-and-delete operation and now only 1GB free before I started the
compression?

BLARGH this is bugging me.

I really appreciate the time you're spending on this!

Jim
Andrew Morton replied on 09-Oct-08 10:04 AM
You could try the freeware version of SpaceMonger to see if something
becomes obvious:

http://www.werkema.com/software/index.php

Andrew
JaimeZ replied on 09-Oct-08 10:05 AM
The "Disk Cleanup Wizard" decided it could free up 1GB by compressing old
files. That was it. No help there.

Somebody else suggested I use "filemon" to see what's going on so I have
that and I'm going to experiment with that shortly.
JaimeZ replied on 09-Oct-08 12:08 PM
I have a new working theory, please tell me if this makes sense?

* When I began this project, the 4.77TB drive was only about half full and
several directories were compressed. I wanted to speed things up so I
unchecked "Compress files in this folder" and let it work for a while. I
didn't notice much change in the free space and that made sense to me because
as I mentioned before, most of these files aren't particularly compressible
to start with.

When I copied the files from one place to another, it may have decompressed
them, such that while file 01CYEE2B.I42 was 284kb in /1/ and /2/, in fact in
/1/ it was semi-compressed and in /2/ it was not, so the files in /2/ took up
more space on the drive.

What I still don't get is why as I delete hundreds of GB out of /1/, the
free space doesn't increase much at all? Is it "expanding" previously
compressed files to fill up the available room? If so, why doesn't it take up
the last available (now) 1.10GB?

When I look at TreeSize, it tells me I have nearly 7TB of file data on the
drive, even though it's only a 4.77TB drive. Several folders that I have
previously un-checked "compress files in this folder" do not show up "blue"
in Explorer but show up "blue" in TreeSize, and indeed if I right-click ->
properties them, they appear to be still compressed by about 1/3.

So is my problem that although I'm deleting hundreds of GB, the remaining
2.5TB "overage" is decompressing to fill up the space?

Thanks again, this is very interesting but frustrating at the same time.

Jim
Pegasus \(MVP\) replied on 09-Oct-08 12:32 PM
You write "When I look at TreeSize, it tells me I have nearly 7TB of
file data on the drive, even though it's only a 4.77TB drive." This is
exactly the phenomenon I predicted when I guessed that you were
dealing with a compressed file issue. I think you're on the right path
now - you just need to dig a little deeper.
JaimeZ replied on 09-Oct-08 12:46 PM
Sure. I apologize for doubting you; it's just that Explorer didn't show any
folders "blue" anymore, so I assumed the files had been decompressed already.

What I'm planning to do is re-enable compression on the larger (and
less-used) folders to see if that helps my situation. Otherwise, am I correct
in assuming that I'd have to delete 2TB of data before I saw my free space
start to increase again??

Thanks for all your help.

Jim
Pegasus \(MVP\) replied on 09-Oct-08 01:01 PM
Your previous report showed that your free disk space increased each time
you deleted some files. It simply did not increase as much as you expected,
because the files you deleted were consuming far less disk space than you
thought they did.
JaimeZ replied on 09-Oct-08 02:00 PM
Interesting how that could be the case without the files appearing blue in
explorer, but now that I have compression running on some folders I've seen
the free space tick up one whole GB already! 1999 to go...

Thanks again for everything!

Jim
R. C. White replied on 10-Oct-08 09:12 AM
Hi, Jim.

Have a look at Folder Options | View.  On about the second page there is a
check box for "Show encrypted or compressed NTFS files in color".  Is this
box checked?

RC
--
R. C. White, CPA
San Marcos, TX
rc@grandecom.net
Microsoft Windows MVP
(Running Windows Live Mail 2008 in Vista Ultimate x64 SP1)
Michael D. Ober replied on 10-Oct-08 11:10 AM
Another way to check the compression status on a directory is to right click
the directory and select properties.  If the size on disk is smaller than
the size of the files, you have compression turned on.  Deleting the
directory will free up at most, the size on disk value.  I say at most,
because deleting through Explorer almost always results in the file simply
being moved to the recycle bin.  To bypass the recycle bin, either use the
command line or hold a shift key down while you do the delete.

Mike Ober.
JaimeZ replied on 10-Oct-08 06:07 PM
Yes, but thanks for your time!
JaimeZ replied on 10-Oct-08 06:08 PM
Right. The mystery is how a directory that has 28GB of files that is maybe
megabyte. I'm positive it's not in the Recyle Bin. ;-)
Michael D. Ober replied on 11-Oct-08 09:02 AM
Jaime,

I think you did this before, but humor me.  Open a command prompt, change to
the directory you want to delete and enter the following commands.

compact > "%userprofile%\desktop\compress.txt"
dir >> "%userprofile%\desktop\bigdir.txt"
dir /ah/ad c:\ >> "%userprofile%\desktop\rootfolders.txt"

On your desktop you will find a files named compress.txt, bigdir.txt, and
rootfolders.txt.  In it will be the current compression information for
every file in the directory.  If the file isn't too large, post it here.  If
it is large, email that file to me at obermd@alum.mit.edu and I'll look at
it.  Something else is going on with these files and we need to see them.  I
know you said you don't have norton, but there are other recycle bin
replacements available and someone may have installed one of them.

Mike.
S replied on 03-Dec-08 11:05 AM
Jim

I have a similar situation. The files are deleted but the free space does
not increase. Were you able to get any resolution
S replied on 03-Dec-08 11:42 AM
On the Drive, from the command prompt, I manually went to RECYLER folder. Its
hidden and the files in it as well. I deleted them to regain the space.
Michael D. Ober replied on 03-Dec-08 07:41 PM
SK,

As far as I know there are no versions of Windows that use "RECYCLER" for
the recycle bin.  They all use Recycle.bin or $Recycle.bin.  In addition,
deleting files from a command prompt (or script), which you did as part of
the testing doesn't send files to the Windows recycle bin.  You don't by any
chance have a third party recycle bin installed?

Mike Ober.
JaimeZ replied on 18-Dec-08 03:59 PM
Mike, it sounds like he went into the RECYCLER folder from the command prompt
and re-deleted the files that way.

SK - no, I never got any resolution there; and "RECYCLER" was definately not
the issue, but thanks for the feedback!

Regards,

Jim