Reading binary files consisting of different data types without a for loop.

55 views (last 30 days)

Adam on 15 Sep 2014

0
Link

Direct link to this question

https://nl.mathworks.com/matlabcentral/answers/154920-reading-binary-files-consisting-of-different-data-types-without-a-for-loop

Edited: Mark Thomson on 2 May 2022

I have a binary data file that consists of some M data sets. Each set of data is made up of Nbytes of a specific template, e.g. [uint16, uint16, uint16, uint32, double, uint32, int16]. Right now I'm just looping over how many data sets I have and reading the information in each data set according to it's type.

for j = 1:Mdatasets
     this(j) = fread(fid,1,'uint32');
     foo(j) = fread(fid,1,'uint16');
     foofoo(j) = fread(fid,1,'double');
     % and so on...
end

Is there a faster way to do this? It can't take a very long time to read some of my larger files (~500MB). I was thinking that if you could give fread() a data type template to repeat over and over like it can do with a single data type, that would be ideal. Not sure if there is a way to do this, or if someone has a way around it, but for loops take so long.

Best Regards,

Adam

1 Comment
Show -1 older commentsHide -1 older comments

Image Analyst on 15 Sep 2014

The for loop is definitely NOT the problem You can do tens of millions of iterations in less than a second and I'm sure you don't have that many files. The time is being taken up by the disk I/O rather than the for loop.

Answers (2)

Guillaume on 15 Sep 2014

2
Link

Direct link to this answer

https://nl.mathworks.com/matlabcentral/answers/154920-reading-binary-files-consisting-of-different-data-types-without-a-for-loop#answer_151787

Edited: Guillaume on 17 Sep 2014

Use the skip argument of fread to read all the elements of the same type at once. From my reading of fread doc, skip is not very straightforward to calculate. This may work:

fieldsizes = [4 2 8 ...]; %uint32 uint16 double ...
skip = @(n) sum([fieldsize(1:n+1) fieldsize(n+1:end)]); %sum up of field sizes except field n
offset = @(n) sum(fieldsizes(1:n)); %offset to element n+1
this = fread(fid, Mdatasets, 'uint32', skip(1));
fseek(fid, offset(1), -1);
foo = fread(fid, Mdatasets, 'uint16', skip(2));
fseek(fid, offset(2), -1);
foofoo = fread(fid, Mdatasets, 'double', skip(3));
fseek(fid, offset(3), -1);
...

edit: added fseek as per Michael comment.

6 Comments
Show 4 older commentsHide 4 older comments

Guillaume on 17 Sep 2014

Thanks Michael for the reminder about the file pointer, I'd completely missed that. It's actually fseek that's needed since you need to go back to the right element.

Reading the file my or Iain's way is bound to be faster than reading it one element at a time.

A third option, probably the fastest is to write a mex file.

Mark Thomson on 2 May 2022

Edited: Mark Thomson on 2 May 2022

Many thanks Guillaume for pointing out the Skip functionality (I should have read the fread help in full first)... I had a similar issue, only with Fortran UNFORMATTED files where this helped skip the record entries.

It seems that the overheads in a for loop are indeed due to the repetitive file access, not being streamlined by the functionality of the compiled built-in fread with Skip.

Image Analyst on 15 Sep 2014

0
Link

Direct link to this answer

https://nl.mathworks.com/matlabcentral/answers/154920-reading-binary-files-consisting-of-different-data-types-without-a-for-loop#answer_151786

You're reading 1 byte at a time - no wonder it takes so long. Read in a whole image at a time:

thisImage = fread(fid, [rows, columns], '*uint16');

5 Comments
Show 3 older commentsHide 3 older comments

Image Analyst on 16 Sep 2014

If you have patterns of data in one file, like a bunch of small variables (header info) and then maybe a big image, then you could make a subroutine to do the "common" part. It could take in a page or slice number and use fseek to go to the starting point for that page/slice. I've written readers for custom image formats, like CT data, and I could give you examples if you want. I read in the header and image data.

Adam on 16 Sep 2014

Right, so you're suggesting skipping around the binary file and reading the 'common' data types like Guillaume suggested.

Products

MATLAB

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!