Linux.com

find /var/www/ -type f >> FullFileList.txt

Link to this post 09 Aug 10

The failure may have been due to a filesize limitation, have you reviewed the size of the output files?

This just came to me, if this indexing is failing because of size then maybe a database solution may be the best bet, it may take a while but it will be easy to search and use.

Link to this post 09 Aug 10

mfillpot wrote:

The failure may have been due to a filesize limitation, have you reviewed the size of the output files?
This just came to me, if this indexing is failing because of size then maybe a database solution may be the best bet, it may take a while but it will be easy to search and use.


no, its stoped between 2 and 3 GB only.
there is TB sized image file on the same partition...

Link to this post 09 Aug 10

joonas wrote:

no, its stoped between 2 and 3 GB only.
there is TB sized image file on the same partition...

With that being the case then I agree with you about a corruption or disk I/O error causing the issue, which will have to be repaired prior to you completing the indexing operation.

On a side note due the massive count of files you are trying to index, a website search can take a while going through a text file. So it went ahead and made a mysql schema and bash script that can be used for an introductory indexing, it takes a while but can be beneficial. The benefits you can get from database indexing would be querying of searches and the ability to code the site to automatically update any new files or deleted files from the database.

Again my script can take a while, but if you are interested I can post the script here.

Link to this post 10 Aug 10

i still wonder if there isnt a way to make the find more error resistant or just continue with the file at a certain line.

it will of course be nice to try you script! :)

Link to this post 12 Aug 10

I will deliver my script and mysql table creation scripts when I have some time available in the next few days.

I personally have never seen find or slocate error out and freeze due to a corruption (I have seen corruptions before and testing indexing of the corrupt files), I am now questioning if ls would even properly display the bad file. I know the apps do have a certain level of error detection and correction, but until we know exactly what is causing the error no mechanism can be build to avoid it. Since you stated the indexing function freeze at the same point I would advise reviewing where it stopped and navigating to the location to see what file or filename is causing the failure.

Link to this post 13 Aug 10

Hello:)
i were now able to complete a filelist with find.

now that the filelist is done i more think about:
creating the database
and maybe to store text/html/css files within the database and also in less soon future ultimatly buying addtional harddrives and do a fast fulltext search......

About your script:
first i thought you already had a completed one, but now iam pretty impressed by your offer to help with a script you are making specialy for this issue! it must not be working already (especially since a part of the issue is gone) but i will anyways be curious to look through the code/ideas you already wrote down.

thanks, Joonas

Who we are ?

The Linux Foundation is a non-profit consortium dedicated to the growth of Linux.

More About the foundation...

Frequent Questions

Join / Linux Training / Board