Orphaned entries on local NTFS HDD keep reappearing

[Laus]

Hi,

I’m seeing persistent orphaned entries on a local NTFS HDD with Everything 1.5 (x64).
My setup:
Local HDD, NTFS, always online , 90,000 files
About 2.5 million files total, including data on servers (indexed via Folders)
For the HDD, the NTFS options are all enabled:
Include in database
Enable USN Journal
Load USN Journal into Recent Changes database
Monitor changes
Automatically remove offline volumes
Maximum size: 32 MB, Allocation data: 8 MB

The USN journal for D: looks normal (fsutil usn queryjournal D: shows a valid range and size).

Symptoms:
When I use “Reindex” on the HDD (context menu in Everything), all orphaned entries disappear.
After that, when I delete files via Python (os.remove(name)), new orphaned entries appear over time: files no longer exist on disk, but still show up in Everything.
I would like to avoid repeated “Force rebuild” or “Reindex” because the total dataset (local + servers) is large and rebuilding takes too long.

Questions:
Under these conditions, what could cause Everything to miss deletions or keep stale entries for the local NTFS HDD, even though USN-based monitoring is enabled?

Is there a recommended way to reset/reinitialize the NTFS/USN indexing for a single volume (the local HDD) without having to fully rebuild the entire database (including server folders)?

Can I trigger a re-indexing in python?

Any Ideas?

Thanks, Laus

[therube]

when I delete files via Python (os.remove(name)), new orphaned entries appear over time

So it is only some times that the deleted file is not removed from Everything's index?

[void]

Everything doesn't detect deleted hard links.
This is a limitation with Everything and the USN Journal.
You will have to reindex your NTFS volume to remove deleted hardlinks.

Is there a recommended way to reset/reinitialize the NTFS/USN indexing for a single volume (the local HDD) without having to fully rebuild the entire database (including server folders)?

To reindex just your NTFS volume:

• In Everything, from the Tools menu, click Options
• Click the NTFS tab on the left.
• Right click your NTFS volume and click Reindex.
• Close the Options window.

-or-

Type in the following search and press ENTER:

/reindex c:

where c: is the NTFS volume to reindex.

Can I trigger a re-indexing in python?

Everything.exe -reindex

Everything.exe -search-command "/reindex c:"

-reindex
-search-command

-or-

With the Everything SDK:

Code: Select all

import ctypes
from ctypes import wintypes

EVERYTHING_IPC_WNDCLASSW = "EVERYTHING_TASKBAR_NOTIFICATION"
WM_USER = 0x0400
EVERYTHING_WM_IPC = WM_USER
EVERYTHING_IPC_REBUILD_DB = 405

user32 = ctypes.windll.user32
user32.FindWindowW.restype = wintypes.HWND
user32.FindWindowW.argtypes = (wintypes.LPCWSTR, wintypes.LPCWSTR)
user32.SendMessageW.restype = wintypes.LPARAM
user32.SendMessageW.argtypes = (wintypes.HWND, wintypes.UINT, wintypes.WPARAM, wintypes.LPARAM)

everything_taskbar_notification_hwnd = user32.FindWindowW(EVERYTHING_IPC_WNDCLASSW, None)

user32.SendMessageW(
    everything_taskbar_notification_hwnd,
    EVERYTHING_WM_IPC,
    EVERYTHING_IPC_REBUILD_DB,
    0
)  # forces all indexes to be rescanned.

Code: Select all

#define EVERYTHING_IPC_WNDCLASSW	L"EVERYTHING_TASKBAR_NOTIFICATION"
#define EVERYTHING_WM_IPC		(WM_USER)
#define EVERYTHING_IPC_REBUILD_DB	405
HWND everything_taskbar_notification_hwnd = FindWindow(EVERYTHING_IPC_WNDCLASS,0);
SendMessage(everything_taskbar_notification_hwnd,EVERYTHING_WM_IPC,EVERYTHING_IPC_REBUILD,0); // forces all indexes to be rescanned.

[Laus]

Everything doesn't detect deleted hard links.

You hit the bull's-eye. There are tons of hard links on this partition.

Thanks a lot, Laus