Thread Subject:

Read specific lines or line from file.

I am working on a system that is reading a large amount of data from text files that are basically sensor recordings at about 25Hz. The way I have been doing this is to read the entire text file with fscanf (or the first N~20000 lines) and just work access them though a cell array at each time step. However this produces an enormousness amount of over head becasue there are about 8 files each with about 50000 lines and 10 columns.

What I was looking to do was to read the files lines by line as I need them. The files will always be accessed in order but the start frame may not always be one (i.e. I might want to start analysis at the 10000th time index). There is also a number of secondary calculations i need to do with each frame from all the files, so i don't want to read directly from the main function.

I would like to either open the file whenever I need to read, and then jump to the line or lines that I would like to read in, and begin there. I haven't found an easy way to do this in matlab yet without reading in the first N lines and just throwing them out...

Any help or suggestions would be greatly appreciated...

-Jameson

On 5/25/2012 3:17 PM, Jameson wrote:
...

> I would like to either open the file whenever I need to read, and then
> jump to the line or lines that I would like to read in, and begin there.
> I haven't found an easy way to do this in matlab yet without reading in
> the first N lines and just throwing them out...
...

Well, the thing about sequential files is that they're well, "sequential".

If they are of fixed length records you can use fseek to move the file
pointer so many bytes and then fscanf() or fgetl() from there, but if
they're variable length then you have a problem in knowing for certain
you got the right number of records unless you have a timestamp or other
identifying data in the record of choice.

If they're not terribly large, another alternative that could
potentially be faster would be to use fread() instead on chunks (or the
whole file) and then do things like search for the nth \n in memory to
get to the right place and then process in memory.

All in all, if you want random-access files you'll need to create the
files in a form that supports such things or a database structure
instead of flat files.

--

On 5/25/2012 4:46 PM, dpb wrote:
> On 5/25/2012 3:17 PM, Jameson wrote:
> ....
>
>> I would like to either open the file whenever I need to read, and then
>> jump to the line or lines that I would like to read in, and begin there.
...

> If they are of fixed length records you can use fseek to move the file
> pointer so many bytes and then fscanf() or fgetl() from there,...

One other alternative--if you're opening the file and know the record
number, textscan() w/ the 'headerlines' option to skip that many records
at the beginning _might_ be somewhat faster than the explicit fgetl()
loop since it's in the compiled support routines rather than a m-file.
But, w/ jit compilation, it's possible the savings may not be that much;
you'd just have to try it and see...

--