[Solved]Making sure deleting works

[Herkules97]

I sometimes come across a block when deleting larger amount of files. I don't know what it is but I can select 200K files and press delete and an hour later let's say it asks if I want to delete via the Windows Explorer prompt.

Other times, I have to delete an unknown amount of files, let's say 20K in 1-2K batches. Selecting all 20K might not work, but selecting 2K at a time does even though all the files will eventually be done. It doesn't seem related to file path length as I even went down to 200 length and it still fails and again the files are able to be deleted in 1-2K batches.

After that I might be able to select the remaining 180K and it works fine to delete whenever the prompt appears from loading all of that.

The original title was "Use any type of exports from EBV to delete files using its contents" until I realised that EBV does have its own deletion prompt that seems guaranteed to work..When it shows up. Also seems much faster than the Explorer one. Doesn't end up sometimes with Explorer just freezing on 99% because it's removing metadata or whatever from the filesystem.

What I'm wondering then is what sort of export can be used to delete files and what can use said exports to delete files? Like can I with cmd's del tell it to delete all file paths that a .txt points to?

And secondarily, is there a way to force the EBV deletion prompt every time? The Explorer one sucks in comparison.

[void]

Can you give any information of the file that triggers the issue?
Is it a cloud file? a file with limited access?

The slow part is converting a filename to a PIDL and then to a shell item.
This can take hours for 100K files+
A shell item is required for the shell file operation.



You can press ESC to cancel converting a filename to a shell item.
This will also cancel the delete operation.
This might help break out of the block.



Everything can use its own deletion routine if:
It's a permanent deletion (Shift + Delete) and there's a very long path (>259 characters) or a path with trailing dot (.)
This routine just calls DeleteFile and avoids the slow shell file operation.



I will add a file_operation_simple_pidl advanced setting.
When enabled, Everything will just pass a simple filename PIDL to get the shell item.
This will be instant, but might prevent Everything from deleting certain files.
It will be interesting to see the results.



You can export your results under File -> Export.
Not sure what tools are out there and what type of file they want.
Everything has support for exporting to a simple txt file with a filename on each line.
-Set the Save as Type to: TXT Text Files (*.txt)

[Herkules97]

Can you give any information of the file that triggers the issue?
Is it a cloud file? a file with limited access?

I can't give you what triggers it given that it happens with lists of 50K+ files and as I said I can delete everything if it's within 1K-2K. I can't narrow it down. It's just a normal file list. Files on da komputah.

The slow part is converting a filename to a PIDL and then to a shell item.
This can take hours for 100K files+
A shell item is required for the shell file operation.

You can press ESC to cancel converting a filename to a shell item.
This will also cancel the delete operation.
This might help break out of the block.

If I didn't mention, when I press shift+del EBV freezes for a while to load the list and then resumes some minute later having accomplished nothing. This happens sometimes, other times it processes the lists without issues. I have no idea what causes a failure. Maybe it's a bug and if I just shift+del the list over and over it will eventually trigger..I could try that next time I have a deletion list.
Unless I've already tried it before and forgotten..

Everything can use its own deletion routine if:
It's a permanent deletion (Shift + Delete) and there's a very long path (>259 characters) or a path with trailing dot (.)
This routine just calls DeleteFile and avoids the slow shell file operation.

There are files above 260 path-length characters yet the thing does not show up most of the time. Idk why it shows up sometimes and not other times in all my deletions. I always use shift+del, I don't bother with the recycle bin since at least 2023 IIRC.
When it shows up it's much faster, it's like when I delete with 7-Zip. Plus not using Explorer means .lnks don't get delayed if it gets stuck on 99% deleting and I archive .lnks to know when I run anything that gets a .lnk made.

I will add a file_operation_simple_pidl advanced setting.
When enabled, Everything will just pass a simple filename PIDL to get the shell item.
This will be instant, but might prevent Everything from deleting certain files.
It will be interesting to see the results.

For the certain files that don't get deleted, I can just remove those after it's done.
I have experienced it erroring with an error 2 on some files with that deletion option from EBV.
But I think every time it was because the files didn't exist..Either way it was so rare that I can just delete those files with other methods.
A common one that usually works if the file isn't locked by a process is 7-Zip.
Do I recall incorrectly that when it errors, there isn't an option to ignore errors and continue? That probably should be an option, pressing ignore every time for potentially 10 million files would take forever and be unnecessary.
I use exports that I load into a different instance for deleting large lists. This way the live instances aren't frozen.
So even if saying to ignore errors will cause it to be frozen for a lot longer it wouldn't matter.
If at the end it hasn't deleted almost anything, then at least the live instances were useable throughout.

IF it doesn't accomplish much, which has not been my experience, I will look at a third-party program for deleting based on say a .txt of file paths.
I know there is one that can delete problematic files on boot and that uses a drag-n-drop list IIRC. I don't know if that can use an exported .txt of file paths. Maybe there is something similiar. It shouldn't read the files, that would add a massive delay. Plus I don't want to restart to delete files, I want to be there when it deletes larger lists. I am not trusting boot deletions for anything more than 1-10 files that are stuck. Idk if it's even necessary, I may have methods these days that can delete files without needing that delete-on-boot one. Like 7-Zip's delete after compression option. Sometimes that hasn't worked, I think I managed to delete the files in other ways..Can't remember how right now.
I could export .m3u8 and load into foobar2000 and delete from there but that might not load the stills and would take forever and require loading the files first.
I could use flat view in 7-Zip to load the entire folder, hope it doesn't take too long and then delete all from there. This could be a last resort.

[therube]

I can select 200K files and press delete

That's going to be a "soft" delete, no? So going to recycle.bin?

(Otherwise, you'd have to do a hard delete, Shift+DEL - unless you change mapping within Everything itself such that the DEL alone works as a hard delete.)

> when I press shift+del EBV freezes for a while to load the list

> I always use shift+del
> I don't bother with the recycle bin since at least 2023 IIRC

So now I'm confused. Are you hard deleting or soft deleting, or either or, at various times?

I will add a file_operation_simple_pidl advanced setting.

(Not that it matters to me, but) but I'd suspect that would alleviate the "when I press shift+del EBV freezes for a while to load the list". (And even if that freeze wasn't particularly long.) [I might be misinterpreting something here?]



I can tell you, that deleting, wanting to delete 100K files - to recycle.bin, takes a LONG time while Recycle Bin is "Preparing to recycle", going through its' machinations.

(I suspect it was a good 15 minutes or so.)
And I'll note, that all during the time, Everything CPU usage was high, not subsiding until Recycle Bin had finished its doings.

The actual deletion only took 3 minutes.
(Again Everything CPU usage was high. Suppose it would make sense to Pause Updates prior to doing something like this.)

On my end, I was getting ~800 deletes/second.

And then I'll note, Everything a long time (& high CPU), after the fact, to... I'm not quite sure just what (in the Instance that I did not initiate the deletes through)?


Oh.
So even a hard delete (Shift+DEL) attempt, even though not using Recycle Bin, still goes through this "Preparing to delete" stage - which does take forever. (And in this case, Instance used to initial the Shift+DEL uses CPU, where the other instance does not.)

And with that, Error 8007000e?
Oh, that must just be an OOM, or similar*, error cause I'm using x86.
Looks like I ramped-up <sp> to a bit over 1 GB or RAM.
.

Everything - crash on attempt 200K file hard delete attempt.png (32.21 KiB) Viewed 1609 times

.
.
.
Everything, two Instances. One used high CPU after the deletes, one did not?
.

Everything - CPU usage after 100K soft deletes in two separate Instances.png (26 KiB) Viewed 1613 times

OOM, or similar*.
OK, here's your real OOM (& I'm OK with that, cause I'm x86).
.

Everything - OOM messing around with deleting 100Ks of files x86.png (3.41 KiB) Viewed 1595 times

Alright, so what's going on here? Why is it so slow? Is it only because of pidl's?
I sure wasn't expecting that - with a hard delete attempt?
.

Everything - Shift+DEL on 200K causes Preparing.png (16.28 KiB) Viewed 1592 times

[Herkules97]

So now I'm confused. Are you hard deleting or soft deleting, or either or, at various times?

There is no confusion as I have been clear, I have only used shift+del since at least 2023 and any time shit goes in recycle bin it was a mistake or bug.
By shift+del I mean straight to the void.

(Not that it matters to me, but) but I'd suspect that would alleviate the "when I press shift+del EBV freezes for a while to load the list". (And even if that freeze wasn't particularly long.) [I might be misinterpreting something here?]

Freezing is how it works regardless of success or not.
Have you not deleted large lists at once or you use some other method for that?
With EBV anything above maybe 1K files will cause a freeze. Freeze time depends on amount.


For the last stuff, deleting has never taken more RAM really.
I think it happens if you delete within recycle bin. Maybe even just opening it might cause high RAM accumulation.
I recently opened it with 350 items and explorer.exe only went up 15MB private bytes from 470MB to 485MB-ish.
But a day ago I opened it with 1.9K entries and deleted down to 350..My explorer.exe was 8GB private bytes at the end. Probably around 400MB before I started..Not that I checked.
In the past when I used recycle bin I usually restarted explorer.exe when I was done to get rid of the accumulation.

Eventually I gave up using it. Idk why I ever used it, it added inconvenience.
I might move something there, then forgot if it should be deleted and now I have files in the bin sitting there because I'd have to find out if I meant to delete them or not. Sometimes I did not intend to delete them, it was temporary storage I think.
At least that's now in the past and I no longer have to question. The files go straight to the void, no need for doubt.
And if I ever used it for temporary storage, I no longer do that. Seems pointless, just move/copy if needed.

Idk what causes 13% CPU usage but I know it's when deleting or creating lots of files.
Either indexing at all is the cause, it's when the instance has journal on or/and extra property indexing.

[therube]

Freezing is how it works regardless of success or not.
Have you not deleted large lists at once or you use some other method for that?
With EBV anything above maybe 1K files will cause a freeze. Freeze time depends on amount.

Yes. Hence my question:

Alright, so what's going on here? Why is it so slow? Is it only because of pidl's?
I sure wasn't expecting that - with a hard delete attempt?

Other programs, I can select 100K files, Shift+DEL, & they are gone in a matter of seconds.

What was confusing me, was that I was assuming that these 100K files were going to Recycle Bin, & I thought there was overhead with that (causing the long delay).

And that does occur.

But then, when I tried the same, selecting 100K files in Everything (directly, not through a List), & tried Shift+DEL, it too took a LONG time before the prompt for the actual deletion came up. (And likewise, Everything was "busy" all during that long time.)

[void]

Everything 1.5.0.1405a adds an option to set the permanent delete method.

To set Everything to use the built-in permanent delete method:

• In Everything 1.5, from the Tools menu, click Options.
• Click the Advanced tab on the left.
• To the right of Show settings containing, search for:
  delete
• Select: file_operation_permanent_delete_method
• Set the value to: Everything
• Click OK.

By default, only the built-in method is used when there's a very long filename or a file with a trailing .
The Windows shell cannot deal with these files.



Everything 1.5.0.1405a also adds an option to use simple PIDLs for file operations.

PIDLs are required to perform file operations with the shell.
Building a normal PIDL requires disk access and is very slow.
Simple PIDLs are just the filename (and some indexed information like file/folder) and do not touch your disk.

To enable simple PIDLs for file operations and improve file operation performance:

• In Everything 1.5, from the Tools menu, click Options.
• Click the Advanced tab on the left.
• To the right of Show settings containing, search for:
  operation
• Select: file_operation_simple_pidl
• Set the value to: true
• Click OK.

I haven't noticed any negative issues with enabling.
However, please note that file information will be missing from the PIDL, which might result in weird display issues, like files showing the wrong date created in dialogs.
Size, date modified and attributes are pulled from your index.

[Herkules97]

Everything 1.5.0.1405a adds an option to set the permanent delete method.

To set Everything to use the built-in permanent delete method:

• In Everything 1.5, from the Tools menu, click Options.
• Click the Advanced tab on the left.
• To the right of Show settings containing, search for:
  delete
• Select: file_operation_permanent_delete_method
• Set the value to: Everything
• Click OK.

By default, only the built-in method is used when there's a very long filename or a file with a trailing .
The Windows shell cannot deal with these files.

I am now using the first option and it works as I'd hope.
Just curious, did you also make an ignore option for when it runs into errors?
For some reason it can get an error 2 when a file has already been deleted or whatever it is and I don't recall there being an option to ignore further errors until it's done.
Then after that I could manually go through and check what remains.
Maybe the errors I've had is because I use .efu exports to delete and maybe I've already deleted some files previously after making the export but before using it to delete and then forgotten.
But it could also be handy for other errors, if there are any.

[void]

Everything doesn't auto ignore errors.

I did add [Skip] and [Skip All] to all file operation error messages.
So on the first error, just hit skip all.