How can I have an online and offline index?

[cal]

I have many external hard drives that I will often connect and disconnect.

I think there was some way to make it display offline as grey but for my use case, preferably I'll run 2 instances of everything.

1 instance will have an index of all currently connected hard drives, 2nd instance will have an index of all offline hard drives.

If this was possible, do I need to make sure the offline index don't have conflicting drive letters?

[void]

Please try creating an Online and Offline filter:

• In Everything, from the Search menu, click Add to filters...
• Change the Name to:
  Online
• Change the Search to:
  !offline:
  
  (I use !offline: here instead of online: because !offline: will match volumes where the online status is unknown -where as online: will only match volumes Everything knows are online)
• Click OK.
  
• In Everything, from the Search menu, click Add to filters...
• Change the Name to:
  Offline
• Change the Search to:
  offline:
• Click OK.

Filters can be activated from the Search menu, Filter bar (View -> Filter Bar), Filter sidebar (View -> Filters), right clicking the status bar, filter macro or filter keyboard shortcut.
Filters can be organized from Search menu -> Organize filters.

online:
offline:



It is possible to run multiple instances.
I don't recommended multiple instances for your case.
Indexing just offline volumes is not possible.

You could setup your main instance to include only online volumes and your other named instance to include online and offline volumes.
Set your main instance to automatically add new volumes and automatically remove offline volumes under Tools -> Options -> NTFS
Leave both these settings unchecked for your named instance.
It's not very memory efficient to do this.

[therube]

If (& it seems) your offline are relatively static,
so wouldn't opening a separate instance, -read-only, work?

(And then, only periodically update your offline indexes, like when they happen to be online .
Likewise, you wouldn't want your offline Instance to include or automatically include your normal online drives, nor new added drives.)

Everything.exe -read-only -instance MY_OFFLINE_DRIVES

It's not very memory efficient to do this.

I understand duplication of work (by Everything), but otherwise, why & how so is it not very efficient?

[void]

-readonly might lead to stale index.
You would have to occasionally remove -readonly to update the index.

File lists is another option if the offline indexes don't change much.

I understand duplication of work (by Everything), but otherwise, why & how so is it not very efficient?

The work done (CPU usage) would not be noticeable.
Duplicating indexes would waste a bit of RAM.
The extra RAM usage shouldn't matter much if you only occasionally run the offline instance.

[SolarTheory]

I'm currently in a similar situation where I have a bunch of offline volumes and network index in the database file. The data is essentially static and I do not need it loaded most of the time. I want to move them to file lists to reduce the current massive memory footprint and database file size, but I am concerned about losing some indexed data in the process. I had various custom properties indexed at the time they went offline, but I don't remember exactly which ones. I want to avoid dumping the entire list of Everything properties + Windows properties to a file list, just to be sure I got everything.

• Is there a tool or command to query the database for a given offline volume to determine the actual properties/columns that have been indexed with data?

• When exporting file lists, will Everything export all data records of all visible columns of properties that were added to the index (via Options > Properties)? I seemed to recall in issue with exporting data to file lists in the past because I had not scrolled down to force Everything to "retrieve" the data first. Not sure how this applies to offline data.

• EFU vs. CSV to archive static indexes that I will occasionally load into Everything. Is there a preferred format (I'm guessing EFU is preferred for this)

Worst case, I dump all properties to a file list and use PowerShell to parse it and figure out that has data and what does not and delete empty fields via the PowerShell script.

[Herkules97]

I have many external hard drives that I will often connect and disconnect.

I think there was some way to make it display offline as grey but for my use case, preferably I'll run 2 instances of everything.

1 instance will have an index of all currently connected hard drives, 2nd instance will have an index of all offline hard drives.

If this was possible, do I need to make sure the offline index don't have conflicting drive letters?

Is memory the reason you don't want to run just one? As in saving the RAM by only keeping on the instance with known online disks?
If you use NTFS on all the HDDs, then EBV will know they're offline if you add them as that.

If you use the same drive letters, you will have to filter in some other way but I presume you already know this.
Can't do E:\ if you want to only find one of 2 E:\ devices. There is an index number inside of Tools -> Debug - Statistics for that.
The search fsi:[number] will find only files for that, all monitors gets an index number.
If you have lots of folders in an instance, it can take a while to load this list every time. Another reason it's better to sub-folder so you don't have to use fsi:.
As long as you've named them something to stand out, you can know which is which.
The easier however is to put everything in a sub-folder. It's what I started doing with all new HDDs and old ones that still work.
So now I can have B:\B1 and B:\B2. No need to know the FSI at least for any that this applies to.
I think you might be able to make a filter for this if you don't make sub-folders. For example naming the macro B1: or whathaveyou for the first B:\ and B2: for second and the search is fsi:1 and fsi:2 or whatever. Maybe I should do this myself for broken HDDs in my monolithic instance, hmm..

[void]

I had various custom properties indexed at the time they went offline

Make a backup of your Everything.db
You can always load old Everything.db files from File -> Open File List.. then export to EFU, CSV etc..

Is there a tool or command to query the database for a given offline volume to determine the actual properties/columns that have been indexed with data?

There's no command line tool to do this.
This information can be found in Everything under Tools -> Debug -> Statistics.
You can query the known properties with ES (the command line interface to Everything).

When exporting file lists, will Everything export all data records of all visible columns of properties that were added to the index (via Options > Properties)? I seemed to recall in issue with exporting data to file lists in the past because I had not scrolled down to force Everything to "retrieve" the data first. Not sure how this applies to offline data.

Only shown columns are exported.
Make sure all your indexed properties are shown as columns before exporting.

All indexed information is exported.
Only previously gathered unindexed information is exported.

To gather all unindexed information before exporting:

• Select all files (Ctrl + A)
• From the File menu, click Read Extended Information.

(Everything will do this automatically if you enable Tools -> Options -> Advanced -> gather_properties_on_export)

All indexed information in offline files will be exported.
Gathering unindexed information will do nothing for offline files.

EFU vs. CSV to archive static indexes that I will occasionally load into Everything. Is there a preferred format (I'm guessing EFU is preferred for this)

EFU is preferred because it will use the full Filename and full resolution timestamps.
CSV exports information as shown, so Name+Path with timestamps in seconds.



efu_filetime
efu_folder_append_path_separator
gather_properties_on_export
csv_type
csv_filetime

[therube]

-readonly might lead to stale index.
You would have to occasionally remove -readonly to update the index.

(Depending on circumstances), but what I've been doing is to periodically, physically copy the .db from updated index to -read-only Instance, & then restarting the -read-only instance (which essentially updates the -read-only index).

There is no interference in the copying from updated index, & the overwriting of the -read-only .db.

I run this from my -read-only Instance:

Code: Select all

:: BU-HOME.bat - back up Everything 15 .db for use elsewhere
:: SjB 03-15-2024

@echo off

:: save to DISK (rather then just RAM) -instance 15's .db
:: takes a moment, so throw in a pause, & wait, a few sec

everything.exe -instance 15 -save-db
ECHO Saving .db to disk...
pause

SET EVERYTHINGDIR=C:\DEV\Locate\15.1413

COPY /y %EVERYTHINGDIR%\"Everything-15.db"        "Everything-HOME.db"
COPY /y %EVERYTHINGDIR%\"Search History-15.csv"   "Search History-HOME.csv"
COPY /y %EVERYTHINGDIR%\"Run History-15.csv"      "Run History-HOME.csv"

:: skipping Everything-15.ini as i make changes to the layout in HOME version that i want to keep
::    COPY /y %EVERYTHINGDIR%\Everything-15.ini       Everything-HOME.ini
ECHO Files copied...
pause

:: update (freshen, actually)
ARJ32 f E-HOME.arj -jt1 -g?
pause

And a different scenario. Here I'm using the -restart switch with the -read-only Instance.

Code: Select all

Everything -instance 15 -save-db

sleep 3
gecho -n -e \a

COPY /y "Everything-15.db"        "C:\DEV\Locate\MULE.1402\Everything-MULE.1402.db"

:: .db is "static", "not locked", if you will, not written until quit (or -save-db)
:: so in that respect, can simply be overwritten - at any time, & it will remain,
:: static... until.  & if you've opened Everything -read-only, the .db will forever
:: be static, unless you manually "update" it (which this will do...)

:: then, on the MULE end, it is only a matter of restarting Everything, which will
:: then load the .db, some static, or some manually updated version, such that its'
:: "-read-only" mode has been "updated", though not by the -instance MULE, but
:: rather "externally", "manually" (by virtue of -instance 15's -save-db...)

:: & with that, Everything -instance MULE -restart
:: & with, session_store_restore_on_first_new_window=1, set (in .ini)
:: will quit/restart Everything, restoring the Session, "manually" updating...



:: -save-db      will issue a command to the existing Everything.exe to save the
::               database to disk - in the background.
:: -save-db-now  will save the database to disk and not return until this completes.


:: so... i could have used:

:: Everything.exe -instance 15 -save-db-now

:: & avoided having to do the sleep, as -save-db-now would have assured i wasn't
:: copying files before it had finished saving the .db to disk

[ed.opinion]

My use case: I have 5 USB HD's with multiple and partial years of data split across them.

I don't want them plugged in 24/7 so I use metallic sharpie to write size, drive letter, year(s) on the outside.

I want to search them while they are unplugged.

I used FileListCreator to make a *.csv of file names, specific columns / attributes & paths.
https://www.sttmedia.com/filelistcreator (it can save in multiple file formats too)

I'm an average nerd & used MS Access to import 5 *.csv files & made a query & search form, but EVERYTHING is a BEAST by comparison!

tldr; I want a searchable list that EVERYTHING can read to search my USB HD's while unplugged.

EDIT: When I searched the forum, I saw a link to a mid-thread reply & scrolled down... but didn't scroll UP to see the "online !offline", so I'll try to figure that out, but I thought when my drives are unplugged EVERYTHING can't see them.

[void]

If your USB HDDs are NTFS:

• In Everything, from the Tools menu, click Options.
• Click the NTFS tab on the left.
• Uncheck Automatically remove offline volumes.
• Click OK.

Now Everything will keep your USB HDDs in your index when unplugged.



Otherwise, add your USB HDDs as a folder index under Tools -> Options -> Folders.

Everything will keep folder indexes, even when the drive is unplugged.



Everything has File lists
Please try the Everything File List Editor under the Tools menu.
Include your file lists in your index under Tools -> Options -> File lists.

[ed.opinion]

Thank you very much @void. That was easy!
Like everything in the app, I got more than I expected!

My drives are indeed NTFS
UncheckED - Automatically remove offline volumes
Clicked OK
Ejected & unplugged one USB drive
Exited Everything & waited 30 seconds - relaunched - just because...

Ejected USB drive = still shown in my filter side bar, it's searchable & text is dimmed!

Clicked a folder & got:
N:\Movies\
The system cannot find the drive specified

OK, that makes sense...
I used - files: batman to narrow it down saw filenames.
I can write a better search term or "prompt" as I go forward.
I also sorted by the PATH column & this can work for me...

======================================
This made me think I understand why you suggested Folders & I assume those indexes allow drill down:
Otherwise, add your USB HDDs as a folder index under Tools -> Options -> Folders.
Everything will keep folder indexes, even when the drive is unplugged.

I added one whole drive and was asked:
"Add NTFS folder?"
This folder is already indexed under the NTFS tab.

I sounded like I was about to make a mistake, but I did it anyway & it indexed the drive.
I drilled down into folders and sub folders.

I ejected the USB drive & left the app open & watched it dim text before my very eyes!

I drilled into folder / sub folders in sidebar folder view, but clicking in the main search results got:
N:\Movies\
The system cannot find the drive specified

Did the same when I clicked files in the main window - as expected...

OK, that makes sense... I will probably use sidebar folder view & better search terms.

======================================
RE: Everything has File lists
Please try the Everything File List Editor under the Tools menu.
Include your file lists in your index under Tools -> Options -> File lists.

OK, the initial view when using: Tools -> File List Editor left me assuming I would only get these columns:

image.png

I saved the list, but looked for options & saw Edit -> Read extended information, clicked that & saved again.
~ IDK if that was necessary?

Ultimately, I clicked ~ File -> Open File List and all the columns I had previously defined are being shown (used RT CLK any column name -> Add Columns -> Select Property < should be on the View drop down or UI options IMO, but what do I know?)!

I ejected the USB drive & left the app open & the text did not dim. Which is fine.

I drilled into folder / sub folders in sidebar folder view, but clicking in the main search results got:
M:\Photos
The system cannot find the drive specified

As expected...

======================================
I'll have to use this for a bit to see what options I like most, but I expect the 1st / NTFS option is the one I'd use, because it's just there in my filter view with all the drives like normal, then I'd use folder view, but it is nice to have options!

Just a note - when I made my first *.efu & opened it via File -> Open File List ~ I did not notice File -> Close File List so I opened a new tab and closed the file list view. That taught me about TABS! I had been using RT CLK sys tray > NEW WINDOW (which is better for side by side comparison) and now found it's in ~ File -> New Window

Final note: Would I be enlarging my index db file size & impacting speed (8-12-16 TB drives) by using both NTFS and Folder View?

[ed.opinion]

PS: no real need for a reply, I only really had that one question at the bottom...

[void]

I saved the list, but looked for options & saw Edit -> Read extended information, clicked that & saved again.
~ IDK if that was necessary?

It's only necessary for updating stale lists.
If you just created the file list, the extended information (size/dates/attributes) will be up-to-date.

Final note: Would I be enlarging my index db file size & impacting speed (8-12-16 TB drives) by using both NTFS and Folder View?

Adding volumes to your index will increase size and impact performance, but its generally going to be insignificant.
How many files are you indexing? (shown in the status bar on the left when the search is empty)
You would have to add 10,000,000+ files to notice any impact.

[ChrisGreaves]

Final note: Would I be enlarging my index db file size & impacting speed (8-12-16 TB drives) by using both NTFS and Folder View?

Adding volumes to your index will increase size and impact performance, but its generally going to be insignificant.How many files are you indexing? (shown in the status bar on the left when the search is empty)You would have to add 10,000,000+ files to notice any impact.

THEORY ONLY
I suspect that there is a cute little bit of code to be written (Optimization and Control and/or MiniMax theory) to balance ideas/calculations when it comes to
(1) number of offline drives
(2) number of files on each device and
(3) size of file containers on each drive.
Consider 10 USB drives each of about 20 TB capacity holding movies of 2 GB each.

Change the number of external drives from ten down to two
Change the number of files on each drive from one million down to two thousand
Change the size of file containers from 2 GB down to 20 KB
(going from Him with a collection of movies to Me with a collection of Documents)

These three factors alone ought to define a 3D space where the Ideal Solution lies somewhere in that defined space.

But the ideal solution for me (one backup drive, 17,000 DOCument files each of 20 KB) must be markedly different from (say) an institution that harbours 170,000 movie files (MP4) each of 2 GB across 10 of those 12 TB drives.

Yes? or No?

In short Me and My system can be so different from Him and His system that there will be no one-size-fits-all solution.

True? (Y/N)

In short a guide to the use of each defining parameter is the most that we can expect.

Thanks, Chris

[ed.opinion]

RE: ...10,000,000+ files to notice any impact.

Well you said 10 million - so I plugged in 3 more / older drives on a USB hub (one is suddenly unreadable)!
~ I swapped ports on the hub but I'm scared now! 10TB was indexed before, but unreadable now (index exists).
- I will Eject it later & try direct in my mobo USB ports & try some utilities I have.

Anyhoo - 6,333,945 items

However - Folder view is showing my 14TB drive twice & IDK why - or - if it is being counted twice?

LOL - well said @ChrisGreaves - Me and My system can be so different from Him and His system...

[ChrisGreaves]

However - Folder view is showing my 14TB drive twice & IDK why - or - if it is being counted twice?

I have experienced this before, usually because I use (DOS) SUBSTituted drive letters to map folders as drives. Then my previously mapped folders end up being duplicated once I work out how to add these "drives" (Fat, ex-Fat, NTFS?, ...) in the Tools Index dialogue box.
Now that I've mentioned this Void will probably want to rap me on the knuckles. Again (grin)
Cheers, Chris

[ed.opinion]

Can I delete one or both of these "folders" in Everything and not risk my data?

[void]

However - Folder view is showing my 14TB drive twice & IDK why - or - if it is being counted twice?

The drive is most likely already indexed as an NTFS volume under Tools -> Options -> NTFS.
Please remove the duplicated drive from Tools -> Options -> Folders.

Can I delete one or both of these "folders" in Everything and not risk my data

Yes, you can remove a folder from your Everything index under Tools -> Options -> Folders -> <select drive> -> Remove.

Everything doesn't write to your hard drives when indexing.

[ed.opinion]

The drive is most likely already indexed as an NTFS volume under Tools -> Options -> NTFS.
Please remove the duplicated drive from Tools -> Options -> Folders.

[Herkules97]

Idk if I'm blind or never mentioned after my last reply, but adding as folders means it won't know it's offline. Hence why it dims when they're added as NTFS.

The only benefit I've found with folder-monitoring is that it doesn't reach a point where it might want to rebuild the index if you take it offline and online again as it doesn't try to read changes to the device while it was offline or used elsewhere or without an EBV instance for it running.
I use that benefit for larger property-heavy instances.
System crashes can be one of those instances where it will want to rebuild the index..Or at least system crashes that happen after enough changes without saving the db has occurred. For me that's only or almost only the OS device itself due to how many reads and writes take place in a day. So the 2 instances that touch the OS device will usually have to be rebuilt after system crashes.

I believe with NTFS it doesn't read the journal if the instance is on when you plug it in again.
Not that I remember if I ever checked this, but I always exit any instance touching a device before plugging it in again so any possible changes will be read again. This doesn't matter much if either the instance only takes some seconds to rebuild or you don't care about the journal. I try to separate dbs after the journals reach near max capacity.
Either the journal reading is separate from reading file changes or they do both occur while the instances touching those devices were never closed. As in keeping an instance on without the device it touches being plugged in and adding or changing files on that device with say a different computer and then plugging it in again..It will find those new/modified files but not have any new entries in the journal section. I will probably never do such a test and I don't know the answer. Unless I can do a quick one with one of the NTFS USB-sticks I have..Eh.