Search archives

Have a suggestion for "Everything"? Please post it here.
Post Reply
JosephWebber
Posts: 4
Joined: Fri Jan 11, 2013 10:12 am

Search archives

Post by JosephWebber »

I find Everything invaluable.
However, I have a lot of files in archives (RAR, Zip etc) which do not appear.
Any chance of adding the ability (maybe optional) of listing the contents of archives?
void
Developer
Posts: 15096
Joined: Fri Oct 16, 2009 11:31 pm

Re: Search archives

Post by void »

This could be possible with file lists.
Unfortunately, I don't have an easy way to generate an EFU file list of all your rar/zip contents at the moment.

You might be able to do this manually with the file list editor:
  • In "Everything", from the Tools menu, click File List Editor.
  • Drag the contents of each rar/zip file into the File list editor window. (You might need Winrar/Winzip)
  • Save the EFU file list to the desktop.
  • Open the file list with Notepad
  • Replace the temp folder with the correct rar/zip filename (this could be time consuming).
    For example:
    "c:\temp\rar$asd\somefile.txt" to "c:\archive\afolder\azipfile.zip\somefile.txt"
  • In "Everything", from the Tools menu, click Options.
  • Click the File lists tab.
  • Click Add...
  • Open your EFU file list from desktop.
  • Click OK.
Any chance of adding the ability (maybe optional) of listing the contents of archives?
I will consider writing a tool to make an EFU file list from rar and zip contents.
thalek
Posts: 4
Joined: Wed Mar 08, 2017 3:25 am

Re: Search archives

Post by thalek »

I use the 3D rendering software Poser, and as a result, I have thousands of .zip, .7z, .rar, and .sit archives, with literally a million-plus files inside those thousands of archive files. The ability to index those archive files, especially on my network drives, would have tremendous value for me, and anyone who is in a similar position.

Supposedly, Windows 10 can index inside archive files, with the proper settings, but if true, it doesn't seem to extend to my files on the network drives.

Right now, I can use Funduc Software's Search and Replace program to search inside archive files, but it's not really designed for rapid search the way Everything is, and therefore it's a rather sluggish way of achieving this.

I'd like to request the ability to search inside my archives via Everything. The archive contents themselves don't change, so realtime monitoring of changes to archive contents wouldn't be necessary, except in the event a new archive file was added. So, perhaps a once-a-week (or user-selectable interval) update sweep, with the ability to force a manual check if I really have to. That would theoretically allow Everything to retain its usual speed for realtime monitoring of other files, while still keeping the archive inventory reasonably up to date
void
Developer
Posts: 15096
Joined: Fri Oct 16, 2009 11:31 pm

Re: Search archives

Post by void »

I have plans to add zip, rar, 7z and sit support in Everything 1.5.

However with one limitation, It will not search recursively (ie: it will not search inside zip files inside zip files). Everything will just read the directory table, no data will be decompressed.

I am considering supporting searching zip files inside zip files, but it will require plug-in support (I don't want to add zip, rar, 7z and sit decompressors directly in Everything).
therube
Posts: 4580
Joined: Thu Sep 03, 2009 6:48 pm

Re: Search archives

Post by therube »

(I would think that there are "cataloger's" out there that would scan archives?
[It kind of stuck in my head that Cathy did that, but it does not.]

Even 7-zip.exe l * will walk a directory tree listing everything it finds.
Output that to a text file & use a grep to search it.

7-zip.exe l * > catalogue.out
grep -i bginfo.exe catalogue.out > foundfiles
showtx.exe foundfiles

Automate it or whatever.
Maybe even even have it only "update" only new or changed files?

Or do similar, massaging the results in a format that Everything likes, .efu, then open that .efu in Everything.)
thalek
Posts: 4
Joined: Wed Mar 08, 2017 3:25 am

Re: Search archives

Post by thalek »

For myself and myself alone, I don't have very many (fewer than 50) archives that have archives in them, so that works for me, specifically. Being able to scan archived files recursively works better, of course, but if it's not practical, I can live with that limitation. You'll have to hear from other people to know if that's more universally true, though.

I know that there are third party solutions for getting Windows to scan inside archived files, but I don't know if they allow recursion. Nor do I know at the moment if they work with Windows 10. (I should try an experiment to see if Funduc's Search and Replace offers recursion. It appears to completely decompress an archive in order to search for a specific piece of text; in my work-around for Everything, I've been putting in a file name or partial file name as the text to be searched for.)

And TheRube and I agree that the database might be well-served by scanning new archive files. (I'm not certain how scanning changed archived files would be any different than the current real-time monitoring of changed files. I got the impression that it was agreed that such real-time scanning of archive files would slow things down, and archive files, by their nature, are rarely modified. If real-time monitoring of changed archives is impractical, we could compromise on an optional programmable sweep of once a day/week/month.)
xyz123
Posts: 2
Joined: Mon Mar 25, 2013 3:27 pm

Re: Search archives

Post by xyz123 »

void wrote:I have plans to add zip, rar, 7z and sit support in Everything 1.5.

However with one limitation, It will not search recursively (ie: it will not search inside zip files inside zip files). Everything will just read the directory table, no data will be decompressed.

I am considering supporting searching zip files inside zip files, but it will require plug-in support (I don't want to add zip, rar, 7z and sit decompressors directly in Everything).
That's awesome. I'm waiting for this feature for years. I hope Everything 1.5 comes quickly :) That's the only thing lacking in Everything for me. I think that limitation would be ok for most people. It's better than having no archive support at all. Please don't let that limitation stop you from implementing this feature :) We need it and thank you so much for your continuing development.
NotNull
Posts: 5167
Joined: Wed May 24, 2017 9:22 pm

Re: Search archives

Post by NotNull »

This seems like a nice challenge. I'll try to write something to auto-generate such an EFU list.
thalek
Posts: 4
Joined: Wed Mar 08, 2017 3:25 am

Re: Search archives

Post by thalek »

A bit of online research recently, looking for a plug-in or a DLL that would read the contents of archive files indicates that there may be a problem in Windows itself, when it comes specifically to network drives.

If true, this will not help our situation a lot.
NotNull
Posts: 5167
Joined: Wed May 24, 2017 9:22 pm

Re: Search archives

Post by NotNull »

NotNull wrote:I'll try to write something to auto-generate such an EFU list.
Just wrote a proof of concept. It (still) has limitations, but there is also good news: you can open files that are inside an archive directly from Everything!


If you want to test it (Test! Not ready for "production" .. by far ) this is how:

[*] Change the location of 7Z.exe (the tool to inspect the ZIP's; it is part of 7-Zip) to your situation in the script:

Code: Select all

       set UNZIP=c:\Program Files\7-Zip\7z.exe
[*] Create a textfile with the names of the archives you want to include
This is my testset (named ZIPList.txt):

Code: Select all

"T:\TEST Folder\TEST Folder.zip"
"\\Cray1\media\_Parkeer\ZIP2EFU\AutoInstallSoftware.rar"
\\cray1\media\_Parkeer\ZIP2EFU\TEST Folder.zip
"\\cray1\media\_Parkeer\ZIP2EFU\Everything Experiments.zip"
[*] Drag/drop ZIPList.txt (or whatever you called your list of archives) to the script.
This will start a CMD window with output similar to this:
2017-07-30 16_30_10-C__Windows_system32_cmd.exe.png
2017-07-30 16_30_10-C__Windows_system32_cmd.exe.png (8.62 KiB) Viewed 29010 times
The output file has the same name as the input file, but with file extension .EFU.

This is what the content of the EFU looks like (snippet):

Code: Select all

Filename,Size,Date Modified,Date Created,Attributes
"T:\TEST Folder\TEST Folder.zip\Everything.chm.zip",1200043,2017-06-16,, 
"T:\TEST Folder\TEST Folder.zip\Everything.ini",20227,2017-07-17,, 
"T:\TEST Folder\TEST Folder.zip\Kattenfoto.jpg",191644,2017-04-15,, 
"T:\TEST Folder\TEST Folder.zip\Windows PowerShell Desired State Configuration Quick Reference for Windows Management Framework 4.0.pdf",249646,2016-02-07,, 
"\\Cray1\media\_Parkeer\ZIP2EFU\AutoInstallSoftware.rar\runonceex.cmd",1587,2006-05-24,, 
"\\Cray1\media\_Parkeer\ZIP2EFU\AutoInstallSoftware.rar\Test RunOnceEx.cmd",53,2006-05-24,, 
"\\Cray1\media\_Parkeer\ZIP2EFU\AutoInstallSoftware.rar\WASDIT.TXT",736,2006-05-24,, 
"\\Cray1\media\_Parkeer\ZIP2EFU\AutoInstallSoftware.rar\apps.txt",722,2006-05-24,, 
"\\Cray1\media\_Parkeer\ZIP2EFU\AutoInstallSoftware.rar\RunOnce Import.reg",4970,2006-05-24,, 
[*] In Everything: Menu:File > Open File List and there select the just created EFU

Output looks like this:
2017-07-30 16_38_21-Everything.png
2017-07-30 16_38_21-Everything.png (32.06 KiB) Viewed 29010 times
[*] If it is a file that can be opened with Explorer (double-clicking the ZIP, browse to the file in the ZIP and double-click that), you can double-click it in Everyting. The file will be extracted from the zip and opened in the associated program:
2017-07-30 17_15_59-Everything.png
2017-07-30 17_15_59-Everything.png (29.06 KiB) Viewed 29009 times
EDIT: script is in the next message (or here: download/file.php?id=377 )


Remarks
- There are limitations. Actually: too much to describe here. This is just a proof of concept.
- I started out writing this in Powershell, but ran into trouble (limitations of Powershell and/or myself).
And thus rewrote it in CMD. In hindsight, it shouldn't be too hard to convert to Powershell again. Maybe later ..
Last edited by NotNull on Sun Jul 30, 2017 10:34 pm, edited 1 time in total.
NotNull
Posts: 5167
Joined: Wed May 24, 2017 9:22 pm

Re: Search archives

Post by NotNull »

Testing would go a whole lot easier if I actually had attached the script ;-)

Can't add it to the original post anymore (why?), so it's in this message.
Attachments
ZIP2EFU__v0.2.cmd.zip
(846 Bytes) Downloaded 838 times
NotNull
Posts: 5167
Joined: Wed May 24, 2017 9:22 pm

Re: Search archives

Post by NotNull »

thalek wrote:A bit of online research recently, looking for a plug-in or a DLL that would read the contents of archive files indicates that there may be a problem in Windows itself, when it comes specifically to network drives.

If true, this will not help our situation a lot.
Do you know what kind of problems? The basic functions of the posted script seem to handle network locations (be it mapped (X:) or UNC (\\server\share) ) without (extra) issues.

But those are (old) DOS-paths. The first version of the script started out with Win32 paths ("Long UNC"), with paths like \\?\C:\path\to\zipfile, \\?\server\share\etc and \\?\UNC\server\share\etc. Those had problems with network locations (that's why I "stepped down" to \\server\share).
Maybe that's what you are referring to?
GL1zdA
Posts: 1
Joined: Mon Nov 20, 2017 11:03 am

Re: Search archives

Post by GL1zdA »

Since you've mentioned making this functionality available via plug-ins, maybe it would be possible to adapt to Total Commander archive plug-ins for directory reading? This would give Everything the ability to index many archive types by reusing the plug-ins written for TC.
Janus
Posts: 84
Joined: Mon Nov 07, 2016 7:33 pm

Re: Search archives

Post by Janus »

Being one of those people who have more archives around than I am willing to discuss.
I would recommend a simple read the TOC of an archive be the standard.

Then have a triggerable recursive function that does all the archives in a drive or directory tree that the individual user selects.
This way you can have the depth where the individual user needs it, while also making the smallest possible change to the main program.

That might also be the place to put a plug-in interface.
Which would allow an external program to recurse the archive. {Or network drive or online drive or tape drive or other thing that is specific to the setting}.
Then feedback in an efu or csv format to Everything for adding to the master index.

Said external program could then be called if someone does something with a file inside an archive.
This allows the external program to handle anything specific to its own needs, without any changes to Everything.
Which would allow things like a collection of offline disc or tape archives to be indexed.
If you select something offline, the external program prompts you for the needed media.
I have customers with tape, optical and offline shelved drives for storage, all three of which are still used.

Just some thoughts.


Janus.
NotNull
Posts: 5167
Joined: Wed May 24, 2017 9:22 pm

Re: Search archives

Post by NotNull »

Recursive searching would be really useful for the tarballs in tar.gz files ...
shawn8888
Posts: 6
Joined: Thu Nov 20, 2014 3:51 am

Re: Search archives

Post by shawn8888 »

NotNull wrote: Just wrote a proof of concept.
Thanks a lot for the script! It works!
I wish a tool from everything itself and it works faster.
wemaba2426
Posts: 1
Joined: Sat Aug 08, 2020 3:52 pm

Re: Search archives

Post by wemaba2426 »

void wrote: Wed Mar 08, 2017 11:22 am I have plans to add zip, rar, 7z and sit support in Everything 1.5.
Has this feature been implemented in 1.5 yet? I have searched the "Everything 1.5 Alpha" topic for "archive" and "archives" but there were 0 matches.
void
Developer
Posts: 15096
Joined: Fri Oct 16, 2009 11:31 pm

Re: Search archives

Post by void »

zip filenames only at the moment.

7z and rar are still on my TODO list.
therube
Posts: 4580
Joined: Thu Sep 03, 2009 6:48 pm

Re: Search archives

Post by therube »

(7z can do a whole lot more then "just" .7z. It can even do .rar. Though not .sit ;-).
Supported formats:
Packing / unpacking: 7z, XZ, BZIP2, GZIP, TAR, ZIP and WIM
Unpacking only: APFS, AR, ARJ, CAB, CHM, CPIO, CramFS, DMG, EXT, FAT, GPT, HFS, IHEX, ISO, LZH, LZMA, MBR, MSI, NSIS, NTFS, QCOW2, RAR, RPM, SquashFS, UDF, UEFI, VDI, VHD, VHDX, VMDK, XAR and Z.
)
Bluscream
Posts: 4
Joined: Sat Feb 11, 2023 3:16 pm

Re: Search archives

Post by Bluscream »

Image
NotNull
Posts: 5167
Joined: Wed May 24, 2017 9:22 pm

Re: Search archives

Post by NotNull »

Welcome, Bluscream!

What is your question/ remark?
Bluscream
Posts: 4
Joined: Sat Feb 11, 2023 3:16 pm

Re: Search archives

Post by Bluscream »

Hello NotNull, i just wanted to share that i got this working:

Code: Select all

# region SETTINGS

# edit your "Compressed" filter in Everything to change which archives are included
use_everything_1_5_alpha = True
ziplist_file = 'ziplist.efu'
zipcontents_file = 'zipcontents.efu'
path_separator = os.pathsep
log_file = 'ziplist.log'

# endregion SETTINGS
Last edited by Bluscream on Tue Nov 07, 2023 6:39 pm, edited 1 time in total.
horst.epp
Posts: 1332
Joined: Fri Apr 04, 2014 3:24 pm

Re: Search archives

Post by horst.epp »

2Bluscream
Both links are getting a 404 error.
NotNull
Posts: 5167
Joined: Wed May 24, 2017 9:22 pm

Re: Search archives

Post by NotNull »

( i guess this repository is still in private mode)

Everything 1.5 also has an option to search for (partial) filenames in zip and rar archives:

Code: Select all

ext:zip   "c:\this folder\"   container-filenames:everything64
Bluscream
Posts: 4
Joined: Sat Feb 11, 2023 3:16 pm

Re: Search archives

Post by Bluscream »

You're right. Sorry about that. Should be fixed now.

Anyway, i tried 1.5's search but it's painfully slow compared to getting the list of files beforehand so our method is still superior as of now
NotNull
Posts: 5167
Joined: Wed May 24, 2017 9:22 pm

Re: Search archives

Post by NotNull »

There is a way to speed this up by adding this information to the index. That way Everything does not need to go to disk on every search to parse the zip files:
  • Go to Menu:Tools > Options > Indexes > Properties
  • Click the Add button
  • In the search box, type container
  • Select Container Filenames from the list
  • Press OK
  • Configure Include only folders: , Exclude folders: and Exclude files: to your liking
  • Press OK
  • Let Everything index the container filenames
    (progress bar will be shown in the statusbar of the main Everything window)
  • Done.
Now searches for container-filenames: should give instant results.
Bluscream
Posts: 4
Joined: Sat Feb 11, 2023 3:16 pm

Re: Search archives

Post by Bluscream »

I did that but even then when i enter "container-filenames:" as search, it will just spit out the actual archives, not their contents
NotNull
Posts: 5167
Joined: Wed May 24, 2017 9:22 pm

Re: Search archives

Post by NotNull »

For that, add the container-filenames: column to the resultlist:

- Right-click the header of the result list, for example on Name
- container-filenames: will likely be one of the context menu entries (as it is indexed), but if not:
- Select Add columns
- Search for container
- select container-filenames:
- OK
- Hovering the container-filenames: fields will reveal the names in the zxip-file
nemoW
Posts: 5
Joined: Thu Aug 26, 2021 1:00 pm

Re: Search archives

Post by nemoW »

NotNull wrote: Wed Nov 08, 2023 4:05 pm There is a way to speed this up by adding this information to the index. That way Everything does not need to go to disk on every search to parse the zip files:
<...>
Now searches for container-filenames: should give instant results.
Thanks for tip. Is there an option to search file names over the disks AND in archives?
NotNull
Posts: 5167
Joined: Wed May 24, 2017 9:22 pm

Re: Search archives

Post by NotNull »

For example:

Code: Select all

"everything" | container-filenames:"everything"

| = OR
the first part will search for files/folders with everything in the name
the second part will search for files/folders with everything in the name when inside a zip/rar/7z file.


When the text to search for does not contain a space, you can omit the ""
void
Developer
Posts: 15096
Joined: Fri Oct 16, 2009 11:31 pm

Re: Search archives

Post by void »

Thanks for tip. Is there an option to search file names over the disks AND in archives?
Indexing filenames from zip archives is on my TODO list.
(so you can search them as normal filenames)



Please consider a filter to search filenames and archive filenames:
  • In Everything, from the Search menu, click Add to filters....
  • Change the Name to: Filename or Archive Filename
  • Change the Search to: #replaceterms:#(:$param:#):,#(:<#term:|containerfilename:#term:>#):
  • Click OK.
When this filter is active, Everything will search filenames and archive filenames.

Filters can be activated from the Search menu, Filter bar (View -> Filters), right clicking the status bar, filter macro or filter keyboard shortcut.
Bluscream
Posts: 4
Joined: Sat Feb 11, 2023 3:16 pm

Re: Search archives

Post by Bluscream »

void wrote: Wed Nov 08, 2023 10:40 pm Indexing filenames from zip archives is on my TODO list.
(so you can search them as normal filenames)
This would be the ideal solution and then Everything's built-in feature would be on par with our custom solutions
horst.epp
Posts: 1332
Joined: Fri Apr 04, 2014 3:24 pm

Re: Search archives

Post by horst.epp »

I like the searching and indexing of filenames from archives
but I found that nested archives are currently not searched nor indexed
adding Properties Container Filenames.
Are there any plans to support this ?
void
Developer
Posts: 15096
Joined: Fri Oct 16, 2009 11:31 pm

Re: Search archives

Post by void »

I have put on my TODO list to index nested archive files. (eg: zip inside zip)

Thank you for the suggestion.
Post Reply