indexing algorithm

General discussion related to "Everything".
Post Reply
nssang
Posts: 3
Joined: Tue Oct 23, 2018 11:52 pm

indexing algorithm

Post by nssang » Wed Nov 04, 2020 4:23 am

I am a student studying programming. I was amazed at the speed of indexing of everything.
Can you tell me an indexin algorithm or some tips or algorithms to study? I am curious about how you can quickly search through that much data.
Please advise.

void
Site Admin
Posts: 5808
Joined: Fri Oct 16, 2009 11:31 pm

Re: indexing algorithm

Post by void » Wed Nov 04, 2020 8:26 am

Everything creates its database from the NTFS master file table.
The database is stored in memory and saved to disk when you exit Everything.
The database is restored from disk the next time you open Everything.
Everything uses USN Journaling to keep its database up to date.

The database is basically a list of all the file names (in UTF-8), size, date modified and pointers to parent folders.
Indexes are maintained for name, path, size and date modified.

Everything is written entirely in C.

Searches are compiled into byte code and executed.
Everything uses an optimized multi-threaded strstr search on every single filename in the database.

I wrote the Everything database specifically for filenames to be efficient as possible.

raccoon
Posts: 116
Joined: Thu Oct 18, 2018 1:24 am

Re: indexing algorithm

Post by raccoon » Tue Nov 10, 2020 7:48 pm

I see that exported EFU "Everything File List" files are CSV "Comma Separated Value" files.

Can you tell me what the Everything.db "ESDb" file format is? Is this an "Event Stream Database" per https://github.com/customerio/esdb ? Or did you roll your own database?

NotNull
Posts: 2399
Joined: Wed May 24, 2017 9:22 pm

Re: indexing algorithm

Post by NotNull » Tue Nov 10, 2020 9:06 pm

Can'[t find the relevant thread at the moment, but it is "roll your own".

void
Site Admin
Posts: 5808
Joined: Fri Oct 16, 2009 11:31 pm

Re: indexing algorithm

Post by void » Tue Nov 10, 2020 9:08 pm

Here is a little info on the Everything.db:

https://www.voidtools.com/support/everything/db/

Note: this is not up to date for Everything 1.4. However, the structure is still very similar.

Post Reply