SDK Question

If you are experiencing problems with "Everything", post here for assistance.
Janus
Posts: 43
Joined: Mon Nov 07, 2016 7:33 pm

SDK Question

Postby Janus » Tue Feb 13, 2018 6:16 am

A question for void since I was unable to PM.

I am using the SDK with a currently private fork of explorer++.
So far I am using it for replacing the foldersize calculations, and will be adding more later.
However, folders with large file counts are still taking quite a while.
I have 8M+ files in my projects drive, it takes a moment.

Is it Okay if I recompile a modified everything32/64.dll from the SDK with a couple added functions, then add it to the downloads of my explorer++ fork?
Sources included of course.


Janus.

void
Site Admin
Posts: 3736
Joined: Fri Oct 16, 2009 11:31 pm

Re: SDK Question

Postby void » Tue Feb 13, 2018 10:06 pm

Yes, that is fine.

The SDK uses the MIT license.

Code: Select all

Copyright (C) 2016 David Carpenter

Permission is hereby granted, free of charge,
to any person obtaining a copy of this software
and associated documentation files (the "Software"),
to deal in the Software without restriction,
including without limitation the rights to use,
copy, modify, merge, publish, distribute, sublicense,
and/or sell copies of the Software, and to permit
persons to whom the Software is furnished to do so,
subject to the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.


Is there something I can change in the SDK to make it faster for you?

Janus
Posts: 43
Joined: Mon Nov 07, 2016 7:33 pm

Re: SDK Question

Postby Janus » Tue Feb 13, 2018 10:42 pm

I actually do not at this moment have an answer.

Right now I am making a query, and getting back a number that more or less represents the total number of results.
Results being files and directories.
I then have to loop through according the result count.
At each step, I check to see if it is a directory, or a file.
If it is a Directory, I increment that count.
If it is a file, I increment that count, and add to the filesize total.
Similar to what I did alphabetizing the loading of addons for celestia in my fork.

A note on my goal here.

I am working on adding a couple of columns to explorer++ which I can use, but others likely won't, though I have been wrong before.

Instead of just foldersize, I also want subfoldercount & subfolderfilecount as well.
These are basically the same as the context menu 'properties' gives you, but in columnar format.
They are not something I will want all the time, but having them available will save me steps when I am shuffling directory trees.

Right now I have to call the loop from inside Explorer++, which adds some overhead.
I am hoping that moving that loop to the DLL, will reduce the overhead.

I return a record with elements I need pretallied.
I then copy the totals to the columns.
More or less like this.
Apologies for the pseudocode, but as I have said, I am not a real C/C++ programmer, I prefer pascal.


folderprops:record
foldercount : uint32;
filecount : uint32;
filesize :uint64;
end;

folderprops GetFolderProps( string FolderPath)
{
folderprops ThisFolder;

Count the files/folders and tabulate them;

ThisFolder.foldercount = foldercount;
ThisFolder,filecount = filecount;
ThisFolder.filesize = tiotalfilesize;

return ThisFolder;
}

Added to the DLL so a single call is made to it, and it calls the everything service/program, saving some overhead I hope.

This is in part to by me time until 1.5 comes out because you had indicated you were considering making these totals available via call, rather than having to brute force and loop through everything from the outside.
If you do include it, then a simple update fixes it, if you do not, then I am prepared.


Janus.

void
Site Admin
Posts: 3736
Joined: Fri Oct 16, 2009 11:31 pm

Re: SDK Question

Postby void » Thu Feb 15, 2018 12:00 am

What about using the folder size in Everything? rather than calculating this yourself.

You can call Everything_GetResultSize on a folder result to get the folder size.

Note: you will need to enable folder size indexing in Everything:
  • In Everything, from the Tools menu, click Options.
  • Click the Indexes tab on the left.
  • Check Index folder size.
  • Click OK.

Janus
Posts: 43
Joined: Mon Nov 07, 2016 7:33 pm

Re: SDK Question

Postby Janus » Thu Feb 15, 2018 12:31 am

The result I got back from that did not correspond to the total size of the folder.

This is what I ended up doing to get the result I wanted.
This is ugly, but it just works, yet it is still faster than HDD/Network access, especially when large numbers of files are involved.
Fair warning though, I am not a real C/C++ programmer, so this may offend purists.

This is in \explorer++\helper\foldersize.cpp in case you want to look at the project.

Code: Select all

HRESULT CalculateFolderSize(TCHAR *szPath,int *nFolders,
int *nFiles,PULARGE_INTEGER lTotalFolderSize)
{
   TCHAR         InitialPath[MAX_PATH + 2];

   DWORD         LastEvError;
   LARGE_INTEGER   MatchSize;
   ULONGLONG      MatchSizeU;
   DWORD         TotResults;
   DWORD         TotIndex;


   StringCchCopy(InitialPath,SIZEOF_ARRAY(InitialPath),_T("\""));
   StringCchCat(InitialPath,SIZEOF_ARRAY(InitialPath),szPath);
   StringCchCat(InitialPath,SIZEOF_ARRAY(InitialPath),_T("\""));

   Everything_SetSearch(InitialPath);

   //EVERYTHING_REQUEST_SIZE                                 (0x00000010)
   Everything_SetRequestFlags(EVERYTHING_REQUEST_SIZE);

   Everything_QueryW(TRUE);

   LastEvError = Everything_GetLastError();
   TotResults = Everything_GetTotResults();

   MatchSizeU   = 0;
   if(TotResults == 0)
      {
      lTotalFolderSize->QuadPart = MatchSizeU;
      return S_OK;
      }


   for (TotIndex = 0; TotIndex < TotResults; TotIndex++)
      {
      Everything_GetResultSize(TotIndex, &MatchSize);

      if (Everything_IsFileResult(TotIndex))
         {
         MatchSizeU = MatchSizeU + MatchSize.QuadPart;
         *nFiles++;
         }
      else
         {
         *nFolders++;
         }
      }

   lTotalFolderSize->QuadPart = MatchSizeU;
      
   return S_OK;
}


I hope this shows the hoops I am jumping through, and I am hoping there are fewer hoops I can use that will still get me the same result.

A couple of additions would be nice.
Everything_GetResultFileCount, Everything_GetResultFolderCount, and Everything_GetResultTotalFileSize would be greatly appreciated.
Or simply make the result database slice returned via in a predefined format so it can be parsed directly without the need for repeated calls.

Perhaps

FileRecord:
Uint32 FoldernameIndex;
strZ Filename;
Uint64 FileSize;
end;

Maybe include a count in a header.
The format doesn't matter all that much, as long as it is documented.
Though being able to directly grab the needed bits with a call would be great.

Janus.

void
Site Admin
Posts: 3736
Joined: Fri Oct 16, 2009 11:31 pm

Re: SDK Question

Postby void » Thu Feb 15, 2018 3:14 am

Please try the following:

Code: Select all

HRESULT GetFolderSize(TCHAR *szPath,PULARGE_INTEGER lTotalFolderSize)
{
   TCHAR         InitialPath[MAX_PATH + 11];
   DWORD         LastEvError;
   LARGE_INTEGER MatchSize;

   // request filelist:"<filename>"
   // this will return the file matching the exact filename.
   StringCchCopy(InitialPath,SIZEOF_ARRAY(InitialPath),_T("filelist:\""));
   StringCchCat(InitialPath,SIZEOF_ARRAY(InitialPath),szPath);
   StringCchCat(InitialPath,SIZEOF_ARRAY(InitialPath),_T("\""));

   Everything_SetSearch(InitialPath);
   
   //EVERYTHING_REQUEST_SIZE                                 (0x00000010)
   Everything_SetRequestFlags(EVERYTHING_REQUEST_SIZE);

   Everything_QueryW(TRUE);

   LastEvError = Everything_GetLastError();

   if (Everything_GetResultSize(0, &MatchSize))
   {
      lTotalFolderSize->QuadPart = MatchSize.QuadPart;
   }
   else
   {
      lTotalFolderSize->QuadPart = 0;
   }
     
   return S_OK;
}


Note the use of the new query:
filelist:"<filename>"

This will find the exact folder by filename.

This does not calculate nFolders or nFiles. Are these values still needed?

Everything_GetResultFileCount
Everything_GetResultFolderCount


There is Everything_GetNumFolderResults and Everything_GetNumFileResults.
However, these functions are no longer work with EVERYTHING_REQUEST_SIZE.
I'll consider adding support for these functions with EVERYTHING_REQUEST_SIZE.

Everything_GetResultTotalFileSize


Added to my TODO list.

Janus
Posts: 43
Joined: Mon Nov 07, 2016 7:33 pm

Re: SDK Question

Postby Janus » Thu Feb 15, 2018 7:25 am

void wrote:Please try the following:

Note the use of the new query:
filelist:"<filename>"

This will find the exact folder by filename.

This does not calculate nFolders or nFiles. Are these values still needed?

Everything_GetResultFileCount
Everything_GetResultFolderCount


There is Everything_GetNumFolderResults and Everything_GetNumFileResults.
However, these functions are no longer work with EVERYTHING_REQUEST_SIZE.
I'll consider adding support for these functions with EVERYTHING_REQUEST_SIZE.

Everything_GetResultTotalFileSize


Added to my TODO list.


I tried the sample code.
Had to match up var names and parameters, but it worked.
Same issues as with mine.
99% cpu usage, and one or two seconds per folder to get size.

I also tried with file: as well.
Same basic result, though it was a little faster, even having to loop through it, and cpu usage stayed at 90% instead of 99%.

Not sure why the difference in return times between the gui with it, and an external call.
It would be nice however, to see that speed outside, or understand what it is from.
If I type in a drive letter 'C:\', it only takes a second to fill in.
If I highlight the 'C' then change it to a 'D', again, it only takes a second to fill it all in.
I was hoping for that sort of speed, or at least with in a magnitude of it.


Janus.

NotNull
Posts: 410
Joined: Wed May 24, 2017 9:22 pm

Re: SDK Question

Postby NotNull » Thu Feb 15, 2018 9:28 am

There might be a completely different approach to get your results faster:

Just (?) do 2 extra Everything queries: c:\windows folder: and c:\windows file:
Those return the total number of results (= #folders/#files) instantaneous (the GUI displays them in the status bar).

EDIT: That should be: c:\windows\ folder: (added a \), to exclude the C:\windows folder itself and to get just the subfolders. And to prevent matching a possible "c:\windows settings" folder.

Janus
Posts: 43
Joined: Mon Nov 07, 2016 7:33 pm

Re: SDK Question

Postby Janus » Thu Feb 15, 2018 4:18 pm

@NotNull

Everything gui response speed is awesome.
My problem is when doing a query from an outside source.

Right now to get the total file size of a query return, it is taking a long time.
I am calling the Everyting DLL from within explorer++ using a quoted path.
I then have to either iterate through the list, or use filelist which self iterates, either of which is magnitudes slower than the native gui is.

The filecount & foldercount are for the future, my only real annoyance is how long it takes to do the redundant calls to get totalfilesize of query result.
Making those two available in explorer++ is part of me making file/folder counts within archives available as columns.
After I have those added, then I am going to add the ability to mount archives as temporary drive letters.
That way they can be worked with easier, but do not pollute the dir tree in the left pane.

I am also hoping to use everything with Reactos when it goes beta, and I can drop M$ windows completely, using the opensource version instead.


Janus.

NotNull
Posts: 410
Joined: Wed May 24, 2017 9:22 pm

Re: SDK Question

Postby NotNull » Thu Feb 15, 2018 5:55 pm

OK, I thought you wanted the file/foldercount as well, as you said:

I then have to loop through according the result count.
At each step, I check to see if it is a directory, or a file.
If it is a Directory, I increment that count.
If it is a file, I increment that count, and add to the filesize total.


Total size should be trivial: let Everything index that (Menu:File > Options > Indexes: enable Index Folder size
(like @void already suggested)


What I would do: (and I'm not a programmer, so be aware of stupid ideas)

- If these results don't match up with the reults you get by stepping through the folders, analyze the differences. (probably hardlinks/ symlinks/etc). Maybe the Everything results are even better for your case? Or find a fast way to get the "different files" and measure those seperately. Add the result to the total.

- WizTree also has an MFT based file/folder count and -size column. Do these results match your's or Everything's? If they match your's: it should be doable in some way (WizTree is also very fast)

- I could live with a filemanager that doesn't show a 100% accurate foldersize (somewhere else on these forums was a discussion about actual filesize. TL;DR: you just don't know). Especially if the alternative would mean a slow filemanager.

void
Site Admin
Posts: 3736
Joined: Fri Oct 16, 2009 11:31 pm

Re: SDK Question

Postby void » Fri Feb 16, 2018 4:11 am

99% cpu usage, and one or two seconds per folder to get size.


Are you calling this for one folder? or for many results?
Each GetFolderSize call will take about ~0.1 seconds.
I would expect about 1-2 seconds with 99% cpu usage for 20 calls to GetFolderSize.

A better solution will be to do a Everything query with filelist:<filename1;filename2;filename3;...> for all your results.
And then researching inside those results for the folder size.

There needs to be a single call, like the Everything_GetRunCountFromFileName but for sizes, and dates. This would bypass the query and just get the information directly from the Everything database. Added to my TODO list.

Janus
Posts: 43
Joined: Mon Nov 07, 2016 7:33 pm

Re: SDK Question

Postby Janus » Fri Feb 16, 2018 7:35 am

@void

In explorer++ using the code I posted, or the adapted code you had me try.
Each folder still takes ~0.5-1.0 seconds to get a size, then the next gets the size set.
Twenty subfolders showing in detail view will take ~10-20 seconds to fill in.
More likely toward the 10, but 20 happens sometimes.

I can post timing samples from the debug window for three variations tomorrow to show what I mean in more detail.

Most of them however, look about like this.

    IPC: execute query: filelist:"C:\Windows"
    search 'filelist:"C:\Windows"' filter '' sort 0
    IPC: wait 2
    term 0078e0d0, flags: 3140, next: 00000000, notnext: 00000000
    folderop: 385, fileop: 385, term: filelist:C:\Windows
    SET SORT 0
    found 1 folders, size 65540, db search time taken: 0.060724 seconds
    found 0 files, size 0, db search time taken: 0.580731 seconds
    set sort 0 1
    already sorted
    finished sort, time taken 0.001389 seconds
    update selection 0.000003 seconds
    DB_WAIT: _db_ready_proc waiting...
    DB_WAIT: _db_ready_proc waited 0.000684 seconds
    when ready 11 0000002d 00000000
    when ready 11 0000002d 00000000
    SIZE CHECK 36 36 diff 0 count 1
    IPC: query complete: 4294967295 max results, offset 0, reply hwnd 12911706
    IPC: watch awake
    IPC: Client disconnected.

or

    IPC: execute query: filelist:"C:\$RECYCLE.BIN"
    search 'filelist:"C:\$RECYCLE.BIN"' filter '' sort 0
    IPC: wait 2
    term 04c483d8, flags: 3140, next: 00000000, notnext: 00000000
    folderop: 385, fileop: 385, term: filelist:C:\$RECYCLE.BIN
    SET SORT 0
    found 1 folders, size 65540, db search time taken: 0.060313 seconds
    found 0 files, size 0, db search time taken: 0.566090 seconds
    set sort 0 1
    already sorted
    finished sort, time taken 0.001465 seconds
    update selection 0.000002 seconds
    DB_WAIT: _db_ready_proc waiting...
    DB_WAIT: _db_ready_proc waited 0.000693 seconds
    when ready 11 0000002d 00000000
    when ready 11 0000002d 00000000
    SIZE CHECK 36 36 diff 0 count 1
    IPC: query complete: 4294967295 max results, offset 0, reply hwnd 15598682
    IPC: watch awake
    IPC: Client disconnected.

When using

Code: Select all

HRESULT CalculateFolderSize(TCHAR *szPath,int *nFolders,
int *nFiles,PULARGE_INTEGER lTotalFolderSize)
{
   TCHAR         InitialPath[MAX_PATH + 11];
   DWORD         LastEvError;
   DWORD         TotIndex;

   LARGE_INTEGER   MatchSize;

   StringCchCopy(InitialPath,SIZEOF_ARRAY(InitialPath),_T("filelist:\""));
   StringCchCat(InitialPath,SIZEOF_ARRAY(InitialPath),szPath);
   StringCchCat(InitialPath,SIZEOF_ARRAY(InitialPath),_T("\""));

   Everything_SetSearch(InitialPath);
   
   //EVERYTHING_REQUEST_SIZE                                 (0x00000010)
   Everything_SetRequestFlags(EVERYTHING_REQUEST_SIZE);

   Everything_QueryW(TRUE);                                                                 

   LastEvError = Everything_GetLastError();

   if (Everything_GetResultSize(0, &MatchSize))
   {
      lTotalFolderSize->QuadPart = MatchSize.QuadPart;
   }
   else
   {
      lTotalFolderSize->QuadPart = 0;
   }
   return S_OK;
}



They are all about like this.
A few are shorter timed, and quite a few are longer.
During that time CPU usage hits ~95-99%

I can compile in debug mode and put it up with symbols if you want to do a profile.
However, the taskmanager clearly shows the high usage happening in everything.

Normal usage of the everything gui does not show any spike.


Janus.

void
Site Admin
Posts: 3736
Joined: Fri Oct 16, 2009 11:31 pm

Re: SDK Question

Postby void » Fri Feb 16, 2018 7:58 am

Thanks for the debug info.
Looks like you have many files.

The main reason for the slowness is Everything is doing a full query for each CalculateFolderSize call.
The full query is completely unnecessary. Unfortunately, I don't have a good solution until Everything 1.5, I will have to implement new API calls for this specific folder size request.

For now, please try searching for:

Code: Select all

   
   TCHAR         InitialPath[MAX_PATH + 24];

   StringCchCopy(InitialPath,SIZEOF_ARRAY(InitialPath),_T("folder: count:1 exact:\""));
   StringCchCat(InitialPath,SIZEOF_ARRAY(InitialPath),szPath);
   StringCchCat(InitialPath,SIZEOF_ARRAY(InitialPath),_T("\""));


folder: will only search for folders. This will help performance as the file database is not touched.
count:1 will only find one result, in some cases this will make the query slower as it limits the search to one thread, however here it should provide a good performance increase.
exact: is similar to filelist:, although slightly faster.

For the best results when you want folder sizes for multiple folders, perform only one Everything query with filelist:"filename1";"filename2";"filename3";...

Janus
Posts: 43
Joined: Mon Nov 07, 2016 7:33 pm

Re: SDK Question

Postby Janus » Fri Feb 16, 2018 3:21 pm

A different approach then.

Since this is being used in a file explorer, it will always be wanting the size of what is to it, a subfolder.
Can I reach upwards to get a wider range of downward content.

For instance, I am browsing C:\
Explorer++ will then request the size of C:\Windows then C:\$Recycle.bin and so forth.

In calculatefoldersize, I have a static variable that is the last query.
When called, it immediately trims a layer off the path.
C:\Windows becomes C:\ for example.
It then compares the new path to the last querypath.
If they match it then loops through the results until the name matches the trimmed name.
In this case it would mean looping through the results of C:\ search for a folder names Windows, than getting its size.

Or I could simply copy the last results to a static cast dynamic array, and if the parent is the same, walk through the array.

Could I use a serch such as 'parent:C: folder:' on a query.
Then loop through the results multiple times?

Or can I do a query on the parent folder of the request, that will give the size of all the subfolders of the parent.
Allowing me to loop through the siblings looking for the one I want?
Then keep that list until the directory is done being looped through?


Janus.

Janus
Posts: 43
Joined: Mon Nov 07, 2016 7:33 pm

Re: SDK Question

Postby Janus » Sat Feb 17, 2018 10:23 pm

Okay, really stupid question here.

I have been trying to take the passed path from explorer++.
C:\Windows for instance.
Get the foldername, Windows in this case.
The parentfolder, C:\ in this case.
Construct a querystring "parent:C:\ folder:", for instance.
Compare that string with Everything_GetSearch().
If they match, use the existing results, if not, do a fresh query.
This should avoid hitting the database so much.

I do not know if it will however, since windows is completely bonkers on string handling.

I found functions to give me the last dir name no problem, and the parent directory name, no problem.

However, thay all want strings in so many different formats that make no sense at all.
TCHAR, LPWSTR, LPCWSTR, And many more.
That leaves the STRSAFE_ fore labelled garbage out of it.

If I try to just use a = b with the strings, a ends up pointing to the same memory location as b, not with a copy, which is insane.
I watched it in debug from VS, incredible.
If I copy the string, which was fun since it seems there is no almost no way to copy between string types.
Then I get told that such and such is uninitialized, so it throws an exception.
Yet when I try to give an initial value, I get 0xc00005 access violations, after being told that TCHAR is not the same as TCHAR[MAX_PATH], and cannot be converted.
Then there is TCHAR vs LPCWSTR vs LPWSTR vs STRSAFE_LPWSTR vs who knows how many different types.

Is there a sane guide out there that shows what formats are what?
One that does not use jargon.
One that just says, here is what it is, and why?
One that tells me why a text in quotes is uncastable/unusable as LPCWSTR or whatever gibberish it spouted this time, so I can concatenate a simple string to use for a query..

And why does it seem like the calls that naturally go together, all require different types of strings that require more work to convert than the job I am trying to do in the first place.
If I do a memory dump, they are the same, so what is the problem, the only difference is in the stupid labels.

All I want is ParentFolder, FolderName, and a way to compare the name of the indexed item for a match so I can use its size.

Why does C/C++ make that so bloody hard?

Just ranting, not really expecting any sanity out of C/C++, but still holding out hope.


Janus.

Janus
Posts: 43
Joined: Mon Nov 07, 2016 7:33 pm

Re: SDK Question

Postby Janus » Sun Feb 18, 2018 9:17 pm

Okay, that was fun ... NOT!!!

I brute forced my way through and made a history check version of the code.

This piece of code works with the idea that I am checking a large number of subdirectories in a folder sequentially.
It derives the name of its parent directory.
It then compares it with the last query done.
If they match, then those results are checked, elsewise, a query is done, and those results are looped through.

My logic behind this was hoping that the subdirctories would have their size as part of the results, they don't.

However, here is the code if someone might find a use for it.

As always, please remember, I am not a real C/C++ programmer.
This code is messy, and brute force, and unoptimized.

Code: Select all

HRESULT CalculateFolderSize(TCHAR *szPath,int *nFolders,
int *nFiles,PULARGE_INTEGER lTotalFolderSize)
{
   TCHAR         InitialPath[MAX_PATH + 2];
   TCHAR   BasePath[MAX_PATH]      = L"";
   TCHAR   MatchFolder[MAX_PATH]   ;
   TCHAR   MatchName[MAX_PATH]      ;
   TCHAR   SearchFolder[MAX_PATH]   = L"";
   TCHAR   MatchSearch[MAX_PATH]   ;
   TCHAR Match_Search[MAX_PATH];


   DWORD         LastEvError;
   LARGE_INTEGER   MatchSize;
   ULONGLONG      MatchSizeU;
   DWORD         TotResults;
   DWORD         TotIndex;

   DWORD SizeError = 0;

   StringCchCopy(BasePath, MAX_PATH, szPath);
   StringCchCopy(MatchName, MAX_PATH, BasePath);


   PathRemoveFileSpec(BasePath);

   StringCchCopy(InitialPath,MAX_PATH,_T("parent:\""));
   StringCchCat(InitialPath, MAX_PATH ,BasePath);
   
   StringCchCat(InitialPath,MAX_PATH, _T("\" folder:"));

   StringCchCopy(MatchSearch,MAX_PATH, Everything_GetSearch());

   if (wcscmp(MatchSearch, InitialPath) != 0)
      {

      Everything_SetSearch(InitialPath);
      Everything_SetRequestFlags(EVERYTHING_REQUEST_SIZE || EVERYTHING_REQUEST_FILE_NAME);
      Everything_QueryW(TRUE);
      LastEvError = Everything_GetLastError();
      }

   TotResults = Everything_GetTotResults();
         SizeError = Everything_GetLastError();

   MatchSizeU   = 0;
   if(TotResults == 0)
      {
      lTotalFolderSize->QuadPart = MatchSizeU;
      return S_OK;
      }

   lTotalFolderSize->QuadPart = 0;

   for (TotIndex = 0; TotIndex < TotResults; TotIndex++)
      {
      StringCchCopy(SearchFolder, MAX_PATH, BasePath);
      StringCchCat(SearchFolder, MAX_PATH, L"\\");
      StringCchCat(SearchFolder, MAX_PATH, Everything_GetResultFileName(TotIndex));
      SizeError = Everything_GetLastError();

      if (wcscmp(SearchFolder, MatchName) == 0)
         {
         SizeError = Everything_GetResultSize(TotIndex, &MatchSize);
         SizeError = Everything_GetLastError();
            MatchSizeU = MatchSize.QuadPart;
         }
      }

   lTotalFolderSize->QuadPart = MatchSizeU;
      
   return S_OK;
}


So, now on to my next idea.


Janus.

Janus
Posts: 43
Joined: Mon Nov 07, 2016 7:33 pm

Re: SDK Question

Postby Janus » Tue Feb 20, 2018 6:18 am

Okay, here is my next attempt.

It generates a parent directory query, then checks to see if the query has changed.
If it has, it updates the query, otherwise, it just proceeds.
Then it loops through the result set.

Response speed is largely determined by total number of files in query.
500K = 1-2 Seconds per directory for updates.
<100K is <1 Sec each.

This one works by comparing file paths, and only adding those in the same tree.

Code: Select all

HRESULT CalculateFolderSize(TCHAR *szPath,int *nFolders,
int *nFiles,PULARGE_INTEGER lTotalFolderSize)
{
   TCHAR   InitialPath[MAX_PATH + 2];
   TCHAR   BasePath[MAX_PATH]      = L"";
   TCHAR   MatchPath[MAX_PATH]   ;
   TCHAR   MatchSearch[MAX_PATH]   ;
   TCHAR   FilePath[MAX_PATH]      = L"";
   TCHAR   File_Name[MAX_PATH]      = L"";


   DWORD         SearchOpts;
   DWORD         LastEvError;
   DWORD         TotResults;
   DWORD         TotIndex;
   LARGE_INTEGER   MatchSize;

   DWORD SizeError = 0;
   DWORD LengthCheck = 0;
   DWORD FLengthCheck = 0;
   DWORD MLengthCheck = 0;
   DWORD MFLengthCheck = 0;
   DWORD FileCount = 0;

   StringCchCopy(BasePath, MAX_PATH, szPath);
   StringCchCopy(MatchPath, MAX_PATH, BasePath);

   PathRemoveFileSpec(BasePath);

   StringCchCopy(InitialPath,MAX_PATH,_T("\""));
   StringCchCat(InitialPath, MAX_PATH ,BasePath);
   
   StringCchCat(InitialPath,MAX_PATH, _T("\" file:"));

   StringCchCopy(MatchSearch,MAX_PATH, Everything_GetSearch());

   
   if (wcscmp(MatchSearch, InitialPath) != 0)
      {

      Everything_SetSearch(InitialPath);
      //EVERYTHING_REQUEST_FILE_NAME                            (0x00000001)
      //EVERYTHING_REQUEST_PATH                                 (0x00000002)
      //EVERYTHING_REQUEST_FULL_PATH_AND_FILE_NAME              (0x00000004)
      //EVERYTHING_REQUEST_EXTENSION                            (0x00000008)
      //EVERYTHING_REQUEST_SIZE                                 (0x00000010)
      //EVERYTHING_REQUEST_DATE_CREATED                         (0x00000020)
      //EVERYTHING_REQUEST_DATE_MODIFIED                        (0x00000040)
      //EVERYTHING_REQUEST_DATE_ACCESSED                        (0x00000080)
      //EVERYTHING_REQUEST_ATTRIBUTES                           (0x00000100)
      //EVERYTHING_REQUEST_FILE_LIST_FILE_NAME                  (0x00000200)
      //EVERYTHING_REQUEST_RUN_COUNT                            (0x00000400)
      //EVERYTHING_REQUEST_DATE_RUN                             (0x00000800)
      //EVERYTHING_REQUEST_DATE_RECENTLY_CHANGED                (0x00001000)
      //EVERYTHING_REQUEST_HIGHLIGHTED_FILE_NAME                (0x00002000)
      //EVERYTHING_REQUEST_HIGHLIGHTED_PATH                     (0x00004000)
      //EVERYTHING_REQUEST_HIGHLIGHTED_FULL_PATH_AND_FILE_NAME  (0x00008000)
      SearchOpts = (EVERYTHING_REQUEST_SIZE | EVERYTHING_REQUEST_PATH | EVERYTHING_REQUEST_FULL_PATH_AND_FILE_NAME);
      Everything_SetRequestFlags(EVERYTHING_REQUEST_SIZE | EVERYTHING_REQUEST_PATH | EVERYTHING_REQUEST_FULL_PATH_AND_FILE_NAME);
      Everything_QueryW(TRUE);
      LastEvError = Everything_GetLastError();
      }

   SearchOpts = Everything_GetResultListRequestFlags();
   TotResults = Everything_GetTotResults();
   SizeError = Everything_GetLastError();

   MatchSize.QuadPart   = 0;
   lTotalFolderSize->QuadPart = 0;
   if(TotResults == 0)
      {
      lTotalFolderSize->QuadPart = MatchSize.QuadPart;
      return S_OK;
      }

   FileCount = 0;
   StringCchCat(MatchPath, MAX_PATH , L"\\");
   MLengthCheck = wcsspn(MatchPath, MatchPath);
   for (TotIndex = 0; TotIndex < TotResults; TotIndex++)
      {
      StringCchCopy(FilePath, MAX_PATH, Everything_GetResultPath(TotIndex));
      StringCchCat(FilePath, MAX_PATH , L"\\");

      MFLengthCheck = _tcsncmp(MatchPath, FilePath,MLengthCheck);
      FLengthCheck = wcsspn(FilePath, FilePath);
      if ((MFLengthCheck == 0))
         {
         SizeError = Everything_GetResultSize(TotIndex, &MatchSize);
         lTotalFolderSize->QuadPart = lTotalFolderSize->QuadPart +  MatchSize.QuadPart;
         //Everything_GetResultFullPathName(TotIndex, File_Name, MAX_PATH);
         FileCount++;
         FileCount--;
         FileCount++;
         }
      }
   return S_OK;
}



As always, remember, I am not a real C/C++ programmer.
I have no doubt there is a more efficient way to accomplish this, I simply don't know it.
In fact, the more I learn, and learn about C/C++, the more I come to dislike them.

Next I am going to add functions to the DLL.

Everything_MakeCache // Takes a path.
Generates query, and compares it to saved query for Cache.
If it does not match, then it does a query on the path.
Creates an array of child directory structures.
Each entry will have Name, FileCount, FolderCount, TotalFileZise, in an array.

Everything_IsParentCached //Checks to see if childname is in CacheArray.

Everything_GetEntry //Takes ChildName, checks if Cached, then returns Cache entry from mathcing index if it is.

Everything_CacheParent //Which will call makecache after deriving it from supplied path.

Everything_ForceGetEntry //Takes path, then creates Parent Cache if needed to get data.


Exact names and breakdown of function will depend on how hard all of this is.


Janus

void
Site Admin
Posts: 3736
Joined: Fri Oct 16, 2009 11:31 pm

Re: SDK Question

Postby void » Thu Feb 22, 2018 11:38 am

Could I use a serch such as 'parent:C: folder:' on a query.
Then loop through the results multiple times?


Yes, that is my recommendation.

Or can I do a query on the parent folder of the request, that will give the size of all the subfolders of the parent.
Allowing me to loop through the siblings looking for the one I want?
Then keep that list until the directory is done being looped through?


Would this only return one result? I'm not sure how you would loop through any siblings this way.
Unless you searched for parent:c:|exact:c: this would include the parent folder in the results.

Is there a sane guide out there that shows what formats are what?


I don't know a guide of the top of my head..
T = ANSI or UNICODE depending on project settings
LP = Long pointer or just a pointer these days.
C = Const (you can not modify the string contents / read only, without this flag assume you can modify the contents)
W = UNICODE
A = ANSI (usually omitted)
CHAR = a single character.
STR = an array of characters.

If there is no T or W flag, assume the string is ANSI.

If you need to convert between ANSI and Unicode use MultiByteToWideChar / WideCharToMultiByte.
Make sure you use _T() around "text" when working with T strings.
Use L"Text" when working directly with W strings
Use "Text" when working directly with A strings.

Usually both A and W string functions are implemented and then T macros are used to point to either.


My logic behind this was hoping that the subdirctories would have their size as part of the results, they don't.


They should, please make sure you request size information:
Change:
Everything_SetRequestFlags(EVERYTHING_REQUEST_SIZE || EVERYTHING_REQUEST_FILE_NAME);
to:
Everything_SetRequestFlags(EVERYTHING_REQUEST_SIZE | EVERYTHING_REQUEST_FILE_NAME);

Good luck!

Janus
Posts: 43
Joined: Mon Nov 07, 2016 7:33 pm

Re: SDK Question

Postby Janus » Thu Feb 22, 2018 5:40 pm

@void

Working in DLL source.

Added a couple of items and two calls.
Take a look at this.

Here is where I am getting results that make no sense to me.
I initially tried with EVERYTHING_REQUEST_PATH, but that gave the directory the folder in question was in.
So I switched to the full path and name, which gave what I wanted.

This is part of my initial attempt, which fails to give folder size.

The point of this is to create a cache of subdirectory names and sizes to a parent.
The reason for this is in explorer++, calls for the sizes sequentially.
So the code checks if the cached directory is the same as its parent.
If not, it makes this call to update the cache.
Once the cache contains the list of the parent's children.
Which is to say the caller and its siblings.
It then looks up the caller, and returns its size.

This allows the first call for directory size to update the cache, and avoid repeated DB queries.
However, I am unable to get folder sizes, so I have added other code to get a file list.
Which I then manually add to the size of the folder where they belong.

Code: Select all

// 180220 Janus Added cachepath and cachedata;
static TCHAR _Everything_CachePath[MAX_PATH] = L" ";

typedef struct EVERYTHING_CacheData
{
   ULARGE_INTEGER FolderSize;
   TCHAR FolderName[MAX_PATH];
}EVERYTHING_CacheData;

static EVERYTHING_CacheData *_Everything_CacheData = NULL;

static DWORD CacheFolderCount = 0;

TCHAR BasePath;

// This is the query built from the path;
// BasePath = 'parent:"C:\\windows" folders:';
BasePath = "parent:\"C:\\windows\" folders:";
Everything_SetSearch(BasePath);
Everything_SetRequestFlags(EVERYTHING_REQUEST_SIZE | EVERYTHING_REQUEST_FULL_PATH_AND_FILE_NAME);
Everything_Query(TRUE);
FeedBack = Everything_GetLastError();

CacheFolderCount = Everything_GetTotResults();
CacheSize = CacheFolderCount * sizeof(EVERYTHING_CacheData);

_Everything_CacheData = _Everything_Alloc(CacheSize);

for (DWORD idx = 0;idx < CacheFolderCount; idx++)
   {
   Everything_GetResultFullPathName( idx, CacheFolderName, MAX_PATH);
   FeedBack1 = Everything_GetResultAttributes(idx);
   FeedBack2 = Everything_GetLastError();
   StringCchCopy(_Everything_CacheData[idx].FolderName, MAX_PATH, CacheFolderName);
   _Everything_CacheData[idx].FolderSize.QuadPart = 0;
   }


When run,

CacheFolderCount is correct.
_Everything_CacheData allocated without error
CacheFolderName is filled in correctly for each loop.

Then the problems start.

FeedBack1 = Everything_GetResultAttributes(idx);

Yields $FFFFFFFF

FeedBack2 = Everything_GetLastError();

Gives Error 8
#define EVERYTHING_ERROR_INVALIDREQUEST 8 // invalid request data, request data first.

Which means the rest is pointless.

My goal is take a path such as C:\windows\inf which is trimmed to its parent, C:\windows
Which is then queried to give all subdirectories, and hopefully their sizes.
I record their names and sizes so when the siblings sizes are queried, it comes from the cache, not the DB.

Doing it this way since the DB query is so time intensive.
Why not just do it once, and save the results for the rest of the calls that will only differ in the final directory name.

If there is something I have missed, please let me know.

If you need to see the whole thing, I can put it an archive on my site.
If you want context, I can include the copy of Explorer++ I am using it with.


Janus.

Janus
Posts: 43
Joined: Mon Nov 07, 2016 7:33 pm

Re: SDK Question

Postby Janus » Fri Feb 23, 2018 8:25 am

Okay, I think I have finally got it.

I need to test further, but I think I finally did it.
Response time is about a second for anywhere up to about 150 folders.
Though it did crash a couple of times, but only after several minutes of continuous movement.

I am also now of the opinion that C/C++ string handling has a gaming alignment of psychotic evil.
I joke, but it certainly feels like a way to try filling insane asylums.

Very few of the string routines, especially of the cmp variety, actually match their documentation.
Also, as for M$ string safe routines, mostly they are not.

The more I learn of C/C++, the less I like either of them.

I will post everything tomorrow if holds up under testing.


Janus.

void
Site Admin
Posts: 3736
Joined: Fri Oct 16, 2009 11:31 pm

Re: SDK Question

Postby void » Fri Feb 23, 2018 8:30 am

Please try Everything_GetResultSize to get the folder size.

Everything_GetResultAttributes will fail, unless you request attributes, for example: Everything_SetRequestFlags(EVERYTHING_REQUEST_SIZE | EVERYTHING_REQUEST_ATTRIBUTES | EVERYTHING_REQUEST_FULL_PATH_AND_FILE_NAME);

This information is not usually indexed, so if you only need to check if the result is a file or folder use the Everything_IsFolderResult/Everything_IsFileResult call instead.

Did the Everything client crash?

Janus
Posts: 43
Joined: Mon Nov 07, 2016 7:33 pm

Re: SDK Question

Postby Janus » Fri Feb 23, 2018 4:45 pm

Okay.

I warn you, this code is messy.
This was structured to make single stepping easier.
I have done no real optimization yet.

The first addition is to everything32.def appended to the end.

Code: Select all

   Everything_GetFolderSize
   Everything_FolderSizeCache


The second is to everything.h

Code: Select all

EVERYTHINGUSERAPI void EVERYTHINGAPI Everything_GetFolderSize(LPCWSTR folderpath, LARGE_INTEGER foldersize );
EVERYTHINGUSERAPI void EVERYTHINGAPI Everything_FolderSizeCache(LPCWSTR cachepath );


The third is in everything.c in two sections.

First:

Code: Select all

// Added to the declarations at the top of the file.

static TCHAR _Everything_CachePath[MAX_PATH] = L" ";

typedef struct EVERYTHING_CacheData
{
   ULARGE_INTEGER FolderSize;
   TCHAR FolderName[MAX_PATH];
}EVERYTHING_CacheData;

static EVERYTHING_CacheData *_Everything_CacheData = NULL;
static DWORD CacheFolderCount = 0;



Second: Just after _Everything_window_proc

Code: Select all


EVERYTHINGUSERAPI void EVERYTHINGAPI Everything_FolderSizeCache(LPCWSTR cachepath ) //Janus testing;
   {
   // Record the path in cachepath;

   // Do search with parent:path folders: to get folder list;
   // allocate cache based on number of folders;
   // Record folder paths in _Everything_CacheData[index]->FolderName;

   TCHAR   BasePath[MAX_PATH];//      = {""};
   TCHAR   CacheFolderName[MAX_PATH];

   DWORD   ItemCount = 0;
   DWORD   CacheSize = 0;
   DWORD   ItemDex = 0;
   DWORD   CachePathLen = 0;
   DWORD   ResultPathLen = 0;
   DWORD   CRPathLen = 0;
   DWORD   FeedBack = 0;
   DWORD   Attribs = 0;
   DWORD   LastError = 0;

   LARGE_INTEGER RCSize;   

   _Everything_Free(_Everything_CacheData);

   StringCchCopy(_Everything_CachePath, MAX_PATH, cachepath);

   StringCchCopy(BasePath,MAX_PATH, L"parent:\"");
   //StringCchCopy(BasePath,MAX_PATH, L"\"");
   StringCchCat(BasePath, MAX_PATH, _Everything_CachePath);
   StringCchCat(BasePath,MAX_PATH, L"\" folders:");

   Everything_SetSearch(BasePath);
   Attribs = EVERYTHING_REQUEST_SIZE | EVERYTHING_REQUEST_FULL_PATH_AND_FILE_NAME;
   Everything_SetRequestFlags(Attribs);
   Everything_Query(TRUE);
   LastError = Everything_GetLastError();

   CacheFolderCount = Everything_GetTotResults();
   CacheSize = CacheFolderCount * sizeof(EVERYTHING_CacheData);

   _Everything_CacheData = _Everything_Alloc(CacheSize);
   for (DWORD idx = 0;idx < CacheFolderCount; idx++)
      {
      if (Everything_IsFolderResult(idx))
         {
         Attribs = 0;
         Everything_GetResultFullPathName( idx, CacheFolderName, MAX_PATH);
         LastError = Everything_GetLastError();
         StringCchCopy(_Everything_CacheData[idx].FolderName, MAX_PATH, CacheFolderName);
         FeedBack = Everything_GetResultSize(idx, &RCSize);
         _Everything_CacheData[idx].FolderSize.QuadPart = RCSize.QuadPart;
         }
      }
   }



EVERYTHINGUSERAPI void EVERYTHINGAPI Everything_GetFolderSize(LPCWSTR folderpath, PULARGE_INTEGER foldersize ) //Janus testing;
   {

   TCHAR   CachePath[MAX_PATH];   
   TCHAR   ParentPath[MAX_PATH];   
   TCHAR   FolderPath[MAX_PATH];   
   TCHAR   CacheHit[MAX_PATH];      

   DWORD   ItemDex = 0;
   DWORD   ItemCmp = 0;

   StringCchCopy(CachePath, MAX_PATH, _Everything_CachePath);
   StringCchCopy(FolderPath, MAX_PATH, folderpath);
   StringCchCopy(ParentPath, MAX_PATH, folderpath);
   PathRemoveFileSpec(ParentPath);
   
   if (lstrcmpi(CachePath, ParentPath) != 0)
      {
      Everything_FolderSizeCache(&ParentPath );
      }

   StringCchCopy(CachePath, MAX_PATH, _Everything_CachePath);
   StringCchCat(CachePath,MAX_PATH, L"\\");

   ItemDex = 0;
   while (ItemDex < CacheFolderCount)
      {
      StringCchCopy(CacheHit, MAX_PATH, _Everything_CacheData[ItemDex].FolderName);
      ItemCmp = lstrcmpi(CacheHit, FolderPath);
      if (lstrcmpi(CacheHit, FolderPath) == 0)
         {
         foldersize->QuadPart = _Everything_CacheData[ItemDex].FolderSize.QuadPart;
         ItemDex = CacheFolderCount;
         }
      ItemDex++;
      }
   }



All of the above is called from CalculateFolderSize inside of /explorer++/helper/foldersize.cpp which is part of the explorer++ project.
That function is called for each subfolder within a directory, serially.
Which is why I made the cache.
The call to getfoldersize checks if its parent is the cache base, if not, it updates the cache.
Cache miss is low because of the nature of the calls.
The first in a series will be a miss, followed by however many, that are cache hits.
Since only the first one with a common parent hits the DB, very low penalty.

Code: Select all

HRESULT CalculateFolderSize(TCHAR *szPath,int *nFolders,
int *nFiles,PULARGE_INTEGER lTotalFolderSize)
   {
   TCHAR folderpath[MAX_PATH];
   StringCchCopy(folderpath, MAX_PATH, szPath);
   Everything_GetFolderSize(szPath, lTotalFolderSize);
   return S_OK;
   }



I have also restructured the SLN to properly name everything32.lib, pdb & dll as part of the build.

Still a work in progress.

Compiler is VS2013 with MBCS installed, but I am not sure if the latter is used.
Many of my problems were string issues, which required single stepping and memory dumps to analyze.

@void
If you wish to use any part of this, either directly or as a guide of any sort.
You are free to do so.
I have tried to follow the pattern of how the rest is done, I hope I have done so.

Once I clean up the code and test it fully, in x86 & x64 both, I will be posting it all on my site.
Though if you implement equivalent functionality, I will use it instead.

Thank you for your patience and help both.
Also, thank you for a very helpful and useful program.

I can post binaries for testing if you like.


Janus.


Return to “Support”