[Solved] Memory leak with many audio, stills and video properties

[Herkules97]

Using 1.5.0.1390a at the very moment

So I began an instance for music stuff for searching outside of foobar2000 with EBV being faster.
This worked fine and dandy when I only scanned my folder named music and then some folders for lyrics. Maybe 285GB total over maybe 70K files.

The plan was always to expand, I already tried by using my monolithic instance first but then I reached max memory without ever reaching a state where I can search anything so I went with folder-based indexing and expanding slowly over time. A bit unsurprising given it has about 180mil files and I was trying to exclude most of them. The sorting stage for the monolith instance itself goes from 23GB to 45GB+ private bytes and that does not have much in the way of exclusions or inclusions. I would probably need hundreds of GBs of RAM to finish..Not that I know how any of it works.

I do not currently know what property is causing an issue, but it seems to also bloat the database file saved. I couldn't save the latest time but the file went from 110MB to 1.9GB which was saved successfully and then for the latest max memory state it got to 10GB .tmp before I force-exited.

I am starting this post now, but won't try stuff until either later today or within the week. I will update to 1404a and see if that helps, if that doesn't help I will be reducing the properties one by one and hope each one doesn't take too long. It might take a few weeks to get through it all, as I have to pay attention. Can't look away for a few seconds or it can max out. I did reach 100% commit charge from the 7 minutes I was on the toilet this one time. Fortunately it did not freeze the whole system for a few minutes or even crash the system.

After I find a state where it stops leaking(I guess by letting it index entirely?), I will revert to all properties except that one and see if it leaks..And then do that over and over until I guess I find the culprit. For all I know I've already had this happen before and then forgotten which tag it is. If so it's probably one of the stills properties. As in still images. That's the only experience I've had with more than the usual 4 I use and IIRC I did have some memory leaking for a property but that was with 1.4x I believe..Or did my laptop have 1.5x..And it was fixed IIRC.

Here are the properties from .ini(I would put these under a spoiler tag so they don't occupy space until you expand but Idk if that's possible with this forum system):

35mm Focal Length,Album,Album Artist,Altitude,Aperture,Artist,Aspect Ratio,Audio Bit Rate,Audio Bits Per Sample,Audio Channels,Audio Format,Audio Sample Rate,Audio Track Count,Author URL,Authors,Beats Per Minute,Bit Depth,Brightness,Camera Maker,Camera Model,Camera Serial Number,Color Representation,Comment,Composer,Compressed Bits Per Pixel,Compression,Conductor,Content Provider,Contrast,Copyright,Date Acquired,Date Released,Date Taken,Description,Digital Zoom,Dimensions,Director,Encoded By,Exif Version,Exposure Bias,Exposure Program,Exposure Time,F-Stop,File Signature,Flash Energy,Flash Maker,Flash Mode,Flash Model,Focal Length,Frame Count,Frame Rate,Genre,Group Description,Height,Horizontal Resolution,Image ID,Initial Key,ISO Speed,Latitude,Length,Lens Maker,Lens Model,Light Source,Longitude,Maker Note,Max Aperture,Media Created,Metering Mode,Mood,Orientation,Parental Rating,Parental Rating Reason,Part of a Compilation,Part of Set,People,Period,Photometric Interpretation,Producer,Promotion URL,Protected,Publisher,QuickTime Metadata,Rating,Related Sound File,Resolution Unit,Saturation,Sharpness,Shutter Speed,Software,Subject,Subject Distance,Subtitle,Subtitle Track Count,Tags,Title,Total Bit Rate,Track,Transcoded For Sync,Vertical Resolution,Video Bit Rate,Video Format,Video Track Count,Vorbis Comment,White Balance,Width,Writer,Year

And here are all the file types I have set them to index, also directly from the .ini(As in every property will index for every filetype below, as I understand this occupies space even if there is no data inside):
"*.3g2;*.3gp;*.3gp2;*.3gpp;*.aac;*.ac3;*.adt;*.adts;*.aif;*.aifc;*.aiff;*.amr;*.amv;*.ape;*.asf;*.asx;*.au;*.avi;*.bdmv;*.bik;*.cda;*.d2v;*.divx;*.drc;*.dsa;*.dsm;*.dss;*.dsv;*.dts;*.ec3;*.evo;*.f4v;*.fla;*.flac;*.flc;*.fli;*.flic;*.flv;*.hdmov;*.ifo;*.ivf;*.lpcm;*.m1a;*.m1v;*.m2a;*.m2p;*.m2t;*.m2ts;*.m2v;*.m3u;*.m3u8;*.m4a;*.m4b;*.m4p;*.m4v;*.mid;*.midi;*.mka;*.mkv;*.mod;*.mov;*.mp2;*.mp2v;*.mp3;*.mp4;*.mp4v;*.mpa;*.mpc;*.mpe;*.mpeg;*.mpg;*.mpls;*.mpv2;*.mpv4;*.mts;*.oga;*.ogg;*.ogm;*.ogv;*.ogx;*.opus;*.pss;*.pva;*.qt;*.ra;*.ram;*.ratdvd;*.rm;*.rmi;*.rmm;*.rmvb;*.roq;*.rpm;*.smil;*.smk;*.snd;*.swf;*.tod;*.tp;*.tpr;*.ts;*.tts;*.uvu;*.vob;*.vp6;*.wav;*.wax;*.weba;*.webm;*.wm;*.wma;*.wmp;*.wmv;*.wmx;*.wvx;*.cue;C:\\Users\\Herku\\Documents\\Foobar2000 ESLyrics\\**.txt;C:\\Users\\Herku\\Documents\\Foobar2000 ESLyrics 0.3.4b1\\**.txt;C:\\Users\\Herku\\Documents\\Foobar2000 Lyric Show\\**.txt;C:\\Users\\Herku\\Documents\\Foobar2000 LyricsS3\\**.txt;C:\\Users\\Herku\\Documents\\Foobar2000 OpenLyrics\\**.txt;*.lrc

Maybe I should use the working db to find what filetypes under my "music" folder don't use certain properties and manually tone down how many it's set to index but that'll take forever. Issue is I have a bunch of files outside of the music folder that I just haven't copied to it. So maybe something outside has stuff in that filetype and property..Maybe it's not worth keeping the property indexed for x filetypes anyway...I just wanted it all, any possible property that could have data for the filetypes above. So if I ever want to search any of the properties, I can. But I could surely do without many of the still image properties.

I have a debug log from one of these instances but I think it may have stopped recording before it really crapped itself. However I was on the toilet for most of it, so it's possible it was already leaking before it stopped recording. It had definitely stopped when it had caused commit to reach 100% as I was back by that point.

[void]

I do not currently know what property is causing an issue, but it seems to also bloat the database file saved. I couldn't save the latest time but the file went from 110MB to 1.9GB which was saved successfully and then for the latest max memory state it got to 10GB .tmp before I force-exited.

One of the string properties could be blowing up.

Monitor a search for:

Code: Select all

len:Album:>65535 | len:Album-Artist:>65535 | len:Artist:>65535 | len:Audio-Format:>65535 | len:Author-URL:>65535 | len:Authors:>65535 | len:Camera-Maker:>65535 | len:Camera-Model:>65535 | len:Camera-Serial-Number:>65535 | len:Comment:>65535 | len:Composer:>65535 | len:Conductor:>65535 | len:Content-Provider:>65535 | len:Copyright:>65535 | len:Description:>65535 | len:Director:>65535 | len:Encoded-By:>65535 | len:Exif-Version:>65535 | len:Flash-Maker:>65535 | len:Flash-Model:>65535 | len:Genre:>65535 | len:Group-Description:>65535 | len:Image-ID:>65535 | len:Initial-Key:>65535 | len:Lens-Maker:>65535 | len:Lens-Model:>65535 | len:Maker-Note:>65535 | len:Mood:>65535 | len:Parental-Rating:>65535 | len:Parental-Rating-Reason:>65535 | len:People:>65535 | len:Period:>65535 | len:Producer:>65535 | len:Promotion-URL:>65535 | len:Publisher:>65535 | len:QuickTime-Metadata:>65535 | len:Related-Sound-File:>65535 | len:Software:>65535 | len:Subject:>65535 | len:Subtitle:>65535 | len:Tags:>65535 | len:Title:>65535 | len:Video-Format:>65535 | len:Vorbis-Comment:>65535 | len:Writer:>65535

You could also try this search without indexing the properties.
Work out which files are causing trouble before adding the property to your index.



I do not recommend indexing all properties.
Properties like QuickTime-Metadata and Vorbis-Comment are huge.

C:\\Users\\Herku\\Documents\\Foobar2000 ESLyrics\\**.txt;...

Use a single

\

here,

\\

will match nothing.

[Herkules97]

I do not currently know what property is causing an issue, but it seems to also bloat the database file saved. I couldn't save the latest time but the file went from 110MB to 1.9GB which was saved successfully and then for the latest max memory state it got to 10GB .tmp before I force-exited.

One of the string properties could be blowing up.

Monitor a search for:

Code: Select all

len:Album:>65535 | len:Album-Artist:>65535 | len:Artist:>65535 | len:Audio-Format:>65535 | len:Author-URL:>65535 | len:Authors:>65535 | len:Camera-Maker:>65535 | len:Camera-Model:>65535 | len:Camera-Serial-Number:>65535 | len:Comment:>65535 | len:Composer:>65535 | len:Conductor:>65535 | len:Content-Provider:>65535 | len:Copyright:>65535 | len:Description:>65535 | len:Director:>65535 | len:Encoded-By:>65535 | len:Exif-Version:>65535 | len:Flash-Maker:>65535 | len:Flash-Model:>65535 | len:Genre:>65535 | len:Group-Description:>65535 | len:Image-ID:>65535 | len:Initial-Key:>65535 | len:Lens-Maker:>65535 | len:Lens-Model:>65535 | len:Maker-Note:>65535 | len:Mood:>65535 | len:Parental-Rating:>65535 | len:Parental-Rating-Reason:>65535 | len:People:>65535 | len:Period:>65535 | len:Producer:>65535 | len:Promotion-URL:>65535 | len:Publisher:>65535 | len:QuickTime-Metadata:>65535 | len:Related-Sound-File:>65535 | len:Software:>65535 | len:Subject:>65535 | len:Subtitle:>65535 | len:Tags:>65535 | len:Title:>65535 | len:Video-Format:>65535 | len:Vorbis-Comment:>65535 | len:Writer:>65535

You could also try this search without indexing the properties.
Work out which files are causing trouble before adding the property to your index.



I do not recommend indexing all properties.
Properties like QuickTime-Metadata and Vorbis-Comment are huge.

C:\\Users\\Herku\\Documents\\Foobar2000 ESLyrics\\**.txt;...

Use a single

\

here,

\\

will match nothing.

I tried the search and at least 15 files that were next to be indexed popped up. All .ogg files.
Vorbis Comment property was the culprit and I've disabled it for ogg but..
I have since compared the properties and found that the synopsis section of vorbis comment does not exist elsewhere? Is that also something that can be made into the description property or can 1 file have both vorbis comment and quicktime metadata?
I haven't gone through much but so far the ogg files that bloated are not. The instance went up to 10s of GBs in RAM within several seconds, now it's staying at 300MB like it should and 15min later it's still at 300MB private bytes.

[void]

Is that also something that can be made into the description property or can 1 file have both vorbis comment and quicktime metadata?

What tag do you want from Vorbis Comment or Quicktime Metadata?

These properties show ALL tags.

[Herkules97]

Is that also something that can be made into the description property or can 1 file have both vorbis comment and quicktime metadata?

What tag do you want from Vorbis Comment or Quicktime Metadata?

These properties show ALL tags.

I added the properties' columns from audio, video and vorbis comment and the only one I couldn't find a duplicate of was the synopsis. The file I used does not have any quicktime metadata so it seems the description is exclusive to it unless there is a property somewhere that has it outside of the 3 property categories I added?

[void]

Ah, sorry, the synopsis tag..

This is not an official Vorbis tag.
Everything doesn't map this tag to any property.

I wonder if it shows up with any Windows Property System property?
Find an ogg file with a synopsis tag.
Right click the result list column header and click Add Columns.
Click the Windows Property System tab on the left.
Right click the result list column header and check Preview.
See if your synopsis tag shows up.

[Herkules97]

Ah, sorry, the synopsis tag..

This is not an official Vorbis tag.
Everything doesn't map this tag to any property.

I wonder if it shows up with any Windows Property System property?
Find an ogg file with a synopsis tag.
Right click the result list column header and click Add Columns.
Click the Windows Property System tab on the left.
Right click the result list column header and check Preview.
See if your synopsis tag shows up.

I don't see it, unless it takes longer to display than my scrolling speed?
This is the URL of the file: https://soundcloud.com/rcxcrafter/knife ... fter-remix

And this is what vorbis comment for it looks like:
title=Knife Party - Death & Desire (RCXcrafter Remix);date=20191119;purl=https://soundcloud.com/rcxcrafter/knife ... nopsis=The Pendulum/Knife Party fan community has collectively decided the Death & Desire remix by Laidback Luke sucked so much that we all needed to make our own remixes. So here's my entry for the contest which is not actually a contest.;DESCRIPTION=https://soundcloud.com/rcxcrafter/knife ... genre=Jazz & Blues;encoder=Lavf58.76.100

Here is an upload of the file, Idk if Pixeldrain removes metadata..If so I can upload an archive with it inside if you want to do tests on it yourself?
https://pixeldrain.com/u/X55BBfp1

[Herkules97]

Find an ogg file with a synopsis tag.

Oh I am stupid, you asked for ogg..But surely any file works? It would still have to apply to this flac file, not just ogg.

Also reading the URL you posted, it says OGG Vorbis. But this is a flac file hmm..I wouldn't know how it all works.
All I know is that there is a field that I don't seem to find elsewhere.

Here is an ogg I narrowed down to causing memory leaking with vorbis comment property: https://pixeldrain.com/u/Mhg9J4HS
In case you want to test that.

[void]

I don't see it, unless it takes longer to display than my scrolling speed?

There's no Windows Property System property that shows the synopsis tag for me.
The Windows Property System is different for each PC. Third party property handlers can be installed.



.flac files can also have vorbis comments.



Thank you for the .ogg file.
The tag after the thumbnail tag in this file is corrupt, which triggers Everything to over allocate an invalid tag.
I will improve handling of corrupt tags in the next update.

[void]

Everything 1.5.0.1405a fixes an issue with parsing large Vorbis comment tags.

There was nothing wrong with your file.

Everything will now also log allocations over 8MB to the debug console and your can check the size of each property index under Tools -> Debug -> Statistics.