| Forum List • Thread List • Refresh • New Topic • Search • Previous • Next  1  |  1. Process large text files#12901 Posted by: PeterJ 2009-05-14 04:59:51 Last edited by: guidance 2010-07-16 21:06:55 (Total edited 2 times) | One of my recent activities was to search through large text files consisting of up to 1 million records with a size of 20 MB. After some experimenting it turned out that to "LOADTEXT" such files into a stem and search in it is too slow. Therefore I implemented a tiny DLL in Purebasic, which reads and process the file within memory, in an amazing speed.
If you like you can try it yourself using the FILELOAD.REX and small sample file of 1500 records. For documentation purposes I also added the source code of the DLL: http://nnnn.3322.org/?f=29711 | 2. Looks really useful #12903 | | One thing that struck me immediately though. Since It's a DLL, do you HAVE to "access" the code via FUNCDEF - you can't achieve the same thing via the library call ? | 3. #12904 Posted by: PeterJ 2009-05-14 16:25:48 Last edited by: PeterJ 2009-05-14 16:28:42 (Total edited 2 times) | | You need to use FUNCDEF, as the I/O parameters need to be defined. The Library directive can only be used for Reginald conform DLLs. This DLL is a standalone one. Concerning performance or flexibility this isn't a disadvantage. | 4. Had another look at your code #12953 | Is there any "documentation" on how it works, what parms it expects etc etc. For example, does FIND allow a PREV as the first argument ? What is the SEM as the second argument for/mean ? If NOCASE is omitted, is the default CASE ?
(It sure as hell is like greased lightning) | 5. #12963 Posted by: PeterJ 2009-05-27 21:54:22 Last edited by: PeterJ 2009-05-27 21:59:02 (Total edited 1 time) | record=FIND("FIRST","SEM","NOCASE")
means return me the record with the first occurrence of search string "SEM".
No PREV is not possible, what do you want achieve?
Brief documentation:
If you FUNCDEF with the same rexx names as I did in the example I posted, we have the following functions:
1. LFILE
records=LFILE(input-text-file)
loads a text file into a memory area and keeps it until the script has ended. The file must be a ASCII text
file, a new line begins after the character combination CR+NL or a simple NL.
If reading of the file was successful the function returns the number of fetched records.
e.g. records=LFILE("C:\TEMP\MYLARGE-FILE.TXT")
2. FIND
record=FIND(mode,search-string,search-option)
mode: <FIRST/NEXT>
if mode=FIRST, it returns the first record where the search string occurs
with NEXT it retrieves the next record, positioned after the previous found record
search-string: string to be found in the fetched records
search-options: <CASE/NOCASE> search is case sensitive (CASE), or insensitive (NOCASE)
The function returns the entire record containing the search string. The initial search must be performed
with the FIRST keyword, any subsequent search must use NEXT.
If search string is not found, or not found after last successful search an empty string is returned.
e.g. record=FIND('FIRST','purple','NOCASE') /* return record with first occurrence of purple in file */
record=FIND('NEXT','purple','NOCASE') /* return record with second occurrence of purple in file */
record=FIND('NEXT','purple','NOCASE') /* return next record with occurrence of purple in file */
if the string is contained more than once in a record, the record is only returned once!
3. GETR
record=GETR(record-number)
returns the content of the requested record. 1 is the first record, 2 the second, etc. If the requested number is not
available, GETR return an emptry string. | 6. Thanks Peter #12977 | No PREV is not possible, what do you want achieve? Nothing as yet - more an understanding of what was possible or not. | | Forum List • Thread List • Refresh • New Topic • Search • Previous • Next 1 |
|
|
搜索
论坛
Home
回顶部
g.yi.org Hits:0
