Custom property indexing default included filetypes
-
- Posts: 192
- Joined: Fri Nov 28, 2014 3:58 pm
Custom property indexing default included filetypes
Hello - It seems when adding a custom property index, the default "include only files" does not include all the possible extensions.
For example, if I choose "width", which is used by pictures and videos, it only includes a portion of the extensions for these categories.
My other question is, if I totally wipe this line out, does that mean it will index ALL relevant filetypes?
Edit: Third kind of related question: Do these custom indexes add a relatively large DB file size and\or RAM Usage load in comparison to the default indexes? Or it will generally use the same resources?
For example, if I choose "width", which is used by pictures and videos, it only includes a portion of the extensions for these categories.
My other question is, if I totally wipe this line out, does that mean it will index ALL relevant filetypes?
Edit: Third kind of related question: Do these custom indexes add a relatively large DB file size and\or RAM Usage load in comparison to the default indexes? Or it will generally use the same resources?
- Attachments
-
- fgwergrewherth.JPG (120.85 KiB) Viewed 24189 times
Last edited by JTCGiants56 on Tue Mar 16, 2021 8:13 am, edited 1 time in total.
Re: Custom property indexing default included filetypes
The default "include only files" is set to the extensions for files that Everything natively supports for the selected property.
I can't think of a good example, but maybe SHA256 hash for files < 1mb?
For example, if you index the length property for *.mp3, only mp3 files in your index will store this property.
The length property needs 8 bytes per file included, so if you do a search for *.mp3, multiply the number of results by 8 will give you the memory usage.
In other words, very little resources are needed.
Yes, if you set clear "include only files", Everything will include ALL files.My other question is, if I totally wipe this line out, does that mean it will index ALL relevant filetypes?
Generally No, if it is a content-based property you may wish to set a limit.Edit: Third kind of related question: Is there a rule of thumb for setting "maximum size"? I'm not exactly sure how big these DBs can get and what will happen if the size it needs for all my files falls below the number.
I can't think of a good example, but maybe SHA256 hash for files < 1mb?
It depends on the property.Do these custom indexes add a relatively large DB file size and\or RAM Usage load in comparison to the default indexes? Or it will generally use the same resources?
For example, if you index the length property for *.mp3, only mp3 files in your index will store this property.
The length property needs 8 bytes per file included, so if you do a search for *.mp3, multiply the number of results by 8 will give you the memory usage.
In other words, very little resources are needed.
-
- Posts: 192
- Joined: Fri Nov 28, 2014 3:58 pm
Re: Custom property indexing default included filetypes
Hmm, so Everything will not support indexing "width" or "height" (aka resolution) for example for .WMV files which is not on this list? If this is a Windows limitation could the MediaInfo app which adds more properties than default be possibly integrated down the line?void wrote: ↑Tue Mar 16, 2021 8:12 am The default "include only files" is set to the extensions for files that Everything natively supports for the selected property.
Yes, if you set clear "include only files", Everything will include ALL files.My other question is, if I totally wipe this line out, does that mean it will index ALL relevant filetypes?
Re: Custom property indexing default included filetypes
Everything will still support width and height for wmv files, it's just not natively supported (yet).
If Everything doesn't understand width/height for a certain file, it will fall back to the Windows Property System.
You can simply add ;*.wmv to your "include only files" to include wmv files.
The default "include only files" is just a suggestion.
"include only files" is not empty by default, as indexing width for everything single file type may take a long time.
I would like to avoid using the Windows Property System by default for performance reasons.
If Everything doesn't understand width/height for a certain file, it will fall back to the Windows Property System.
You can simply add ;*.wmv to your "include only files" to include wmv files.
The default "include only files" is just a suggestion.
"include only files" is not empty by default, as indexing width for everything single file type may take a long time.
I would like to avoid using the Windows Property System by default for performance reasons.
-
- Posts: 192
- Joined: Fri Nov 28, 2014 3:58 pm
Re: Custom property indexing default included filetypes
Ah okay gotcha. I'm sure you've heard of Icaros\Mediainfo which adds properties to filetypes (video bitrate etc) to filetypes that Windows does not show natively. I have no idea if it has an API or whatever but I believe this extra info would be helpful.void wrote: ↑Tue Mar 16, 2021 8:20 am Everything will still support width and height for wmv files, it's just not natively supported (yet).
If Everything doesn't understand width/height for a certain file, it will fall back to the Windows Property System.
You can simply add ;*.wmv to your "include only files".
The default "include only files" is just a suggestion.
"include only files" is not empty by default, as indexing width for everything single file type may take a long time.
I would like to avoid using the Windows Property System by default for performance reasons.
Re: Custom property indexing default included filetypes
Thanks for the suggestions.
A plugin for Mediainfo might be added in a future version of Everything.
Everything will offer native support for common file types.
A plugin for Mediainfo might be added in a future version of Everything.
Everything will offer native support for common file types.
-
- Posts: 192
- Joined: Fri Nov 28, 2014 3:58 pm
Re: Custom property indexing default included filetypes
Sounds good. Thanks for the Info.
It's currently taking several hours to index the properties of my videos\pictures for width\height. Enabled fast sort too. I have a fairly fast pc as well. Is this normal or it might be because I included several non-officially supported filetypes?
It's currently taking several hours to index the properties of my videos\pictures for width\height. Enabled fast sort too. I have a fairly fast pc as well. Is this normal or it might be because I included several non-officially supported filetypes?
Re: Custom property indexing default included filetypes
You can check the indexing progress from Tools -> Options -> Properties.
Is there a filename shown in the top right corner?
Is this filename changing or stuck?
Is there a filename shown in the top right corner?
Is this filename changing or stuck?
-
- Posts: 192
- Joined: Fri Nov 28, 2014 3:58 pm
-
- Posts: 192
- Joined: Fri Nov 28, 2014 3:58 pm
Re: Custom property indexing default included filetypes
I have remove offline drives set. I accidentally turned off 2 data drives and everything had to re-index all my properties for the two which took a bit of time. I'm guessing there's no way around this except for not removing offline drives?
I also have 3 properties being indexed. Is this normal for the Mem usage to be this much higher than 1.4?
I also have 3 properties being indexed. Is this normal for the Mem usage to be this much higher than 1.4?
- Attachments
-
- ever.JPG (46.11 KiB) Viewed 24073 times
Re: Custom property indexing default included filetypes
Everything should not be rescanning properties when adding/removing a volume to your index.
Could you please let me know what is shown in your Tools -> Debug -> Statistics -> Last Rebuild Reason?
You should expect to see about 8 bytes per file per property.
For example: Length for 1 million files will need about 8MB.
Are you using content indexing?
Could you please let me know what is shown in your Tools -> Debug -> Statistics -> Last Rebuild Reason?
Could you please send me your statistics from Tools -> Debug -> Statistics?High memory usage
You should expect to see about 8 bytes per file per property.
For example: Length for 1 million files will need about 8MB.
Are you using content indexing?
-
- Posts: 192
- Joined: Fri Nov 28, 2014 3:58 pm
Re: Custom property indexing default included filetypes
Last rebuild reason: Add NTFS volume.void wrote: ↑Sat Mar 20, 2021 3:53 am Everything should not be rescanning properties when adding/removing a volume to your index.
Could you please let me know what is shown in your Tools -> Debug -> Statistics -> Last Rebuild Reason?
Could you please send me your statistics from Tools -> Debug -> Statistics?High memory usage
You should expect to see about 8 bytes per file per property.
For example: Length for 1 million files will need about 8MB.
Are you using content indexing?
I will send over an email. Content indexing is disabled.
Total Objects: 11,449,366 - Property indexing only set to several video file types (some not officially supported by 1.5 like .wmv)
Re: Custom property indexing default included filetypes
Everything should be reusing the existing properties.Last rebuild reason: Add NTFS volume.
I'll look into this some more..
What properties are you indexing under Tools -> Options -> Properties?
-
- Posts: 192
- Joined: Fri Nov 28, 2014 3:58 pm
Re: Custom property indexing default included filetypes
Just want to reiterate I do have "remove offline volumes" checked. Does it keep the existing properties for these files in the DB even if a drive is removed temporarily?
- Attachments
-
- 4343434.JPG (69.79 KiB) Viewed 24022 times
Re: Custom property indexing default included filetypes
Thanks for the info JTCGiants56,
I'm still looking into the high memory usage issue.
There might be a memory leak that I haven't been able to track down yet..
For 11million files, with those 3 properties, I would expect to see about 1364 MB of memory usage.
~100 MB per million files.
+ ~24 MB per million files for the 3 properties (for the indexed-property-included-files only).
Without indexing properties you should see about 1100 MB of memory usage (the same as Everything 1.4)
Everything 1.5.0.1247 improves memory usage accuracy in Tools -> Debug -> Statistics.
Is Total size close to Everything's memory usage? -if not it may indicate a memory leak.
Does Everything's memory usage grow over time?
Please try removing the Media created property from your index (from Tools -> Options -> Properties).
This is a Windows Property System property.
Everything will look for Windows Property System properties in every single file by default.
For an alternative, please try the Date encoded Everything property. (It's the same property)
I'm still looking into the high memory usage issue.
There might be a memory leak that I haven't been able to track down yet..
For 11million files, with those 3 properties, I would expect to see about 1364 MB of memory usage.
~100 MB per million files.
+ ~24 MB per million files for the 3 properties (for the indexed-property-included-files only).
Without indexing properties you should see about 1100 MB of memory usage (the same as Everything 1.4)
Everything 1.5.0.1247 improves memory usage accuracy in Tools -> Debug -> Statistics.
Is Total size close to Everything's memory usage? -if not it may indicate a memory leak.
Does Everything's memory usage grow over time?
These properties are removed from your index when the volume is removed.Just want to reiterate I do have "remove offline volumes" checked.
Does it keep the existing properties for these files in the DB even if a drive is removed temporarily?
Please try removing the Media created property from your index (from Tools -> Options -> Properties).
This is a Windows Property System property.
Everything will look for Windows Property System properties in every single file by default.
For an alternative, please try the Date encoded Everything property. (It's the same property)
-
- Posts: 192
- Joined: Fri Nov 28, 2014 3:58 pm
Re: Custom property indexing default included filetypes
Found the discrepancy. I've excluded image files in 1.5, just like I have in 1.4 and my object count dropped to 1.7M. I'm also on Everything 1.5.0.1247 now.void wrote: ↑Sun Mar 21, 2021 10:44 am Thanks for the info JTCGiants56,
I'm still looking into the high memory usage issue.
There might be a memory leak that I haven't been able to track down yet..
For 11million files, with those 3 properties, I would expect to see about 1364 MB of memory usage.
~100 MB per million files.
+ ~24 MB per million files for the 3 properties.
Without indexing properties you should see about 1100 MB of memory usage (the same as Everything 1.4)
Everything 1.5.0.1247 improves memory usage accuracy in Tools -> Debug -> Statistics.
Is Total size close to Everything's memory usage? -if not it may indicate a memory leak.
Does Everything's memory usage grow over time?
These properties are removed from your index when the volume is removed.Just want to reiterate I do have "remove offline volumes" checked.
Does it keep the existing properties for these files in the DB even if a drive is removed temporarily?
Please try removing the Media created property from your index (from Tools -> Options -> Properties).
This is a Windows Property System property.
Everything will look for Windows Property System properties in every single file by default.
For an alternative, please try the Date encoded Everything property. (It's the same property)
Now mem usage is almost exact.
- Attachments
-
- er.JPG (17.56 KiB) Viewed 23894 times
-
- Posts: 192
- Joined: Fri Nov 28, 2014 3:58 pm
Re: Custom property indexing default included filetypes
Weird, it started crashing at startup for me after doing the above. I was able to quickly press force rebuild and now its reindexing properties. I'll try to enable debugging if it happens again.
-
- Posts: 192
- Joined: Fri Nov 28, 2014 3:58 pm
Re: Custom property indexing default included filetypes
It's crashing again, but I'm unable to create a dump since it crashes immediately at startup.void wrote: ↑Sun Mar 21, 2021 10:12 pm Thanks for the crash report.
If you see a crash dialog, please send me a Mini crash dump.
I tried going back several beta versions (which worked before), and they are crashing now as well.
So I think It may be related to the excludes I added after installing 1247a.
Code: Select all
*.jpg;*.lnk;*.jpeg;*.torrent;*.symlink;*.png
Re: Custom property indexing default included filetypes
Thank you for the crash report JTCGiants56,
Was there a crash dialog?
Please try the latest Everything 1.5 Alpha.
Version 1.5.0.1248 fixes a crash when a renamed file would be excluded.
Does the crash persist?
Was there a crash dialog?
Please try the latest Everything 1.5 Alpha.
Version 1.5.0.1248 fixes a crash when a renamed file would be excluded.
Does the crash persist?
-
- Posts: 192
- Joined: Fri Nov 28, 2014 3:58 pm
Re: Custom property indexing default included filetypes
No crash dialog.void wrote: ↑Mon Mar 22, 2021 6:54 am Thank you for the crash report JTCGiants56,
Was there a crash dialog?
Please try the latest Everything 1.5 Alpha.
Version 1.5.0.1248 fixes a crash when a renamed file would be excluded.
Does the crash persist?
No crash yet since installing this new version.
Thanks.