Textscan with blocks of data with headers to individual block

2 views (last 30 days)
Raj Raj
Raj Raj on 25 Oct 2014
Answered: Gabriel Felix on 24 May 2020
Hi All,
I am trying to extract to following data with textscan. But I am not able to implement it correctly. Each block consists of 11 data lines and 8 unnecessary lines to exclude.
Data file (testdata.dat) is :
Title
Data Header: Time: 100.16064 Sec 10/8/2012 3:32:17 PM
Data Acquisition: T
Station Name: default
Test File Name:
Axial Axial Axial
segments mm kN
0 -0.19470811 -0.0067140702
0 -0.19806515 -0.0033570339
0 -0.19806515 -0.0033570339
0 -0.19470811 -0.0067140702
0 -0.16785182 -0.0033570339
0 -0.16785182 -0.0067140702
0 -0.13763849 -0.0067140702
0 -0.12421035 -0.0067140702
0 -0.10742517 -0.0033570339
0 -0.11413924 -0.0033570339
0 -0.14099553 -0.0033570339
Data Header: Time: 100.16064 Sec 10/8/2012 3:32:17 PM
Data Acquisition: T
Station Name: default
Test File Name:
Axial Axial Axial
segments mm kN
0 -0.16785182 -0.0033570339
0 -0.20813625 -0.0067140702
0 -0.2316355 -0.0067140702
0 -0.25513476 -0.0067140702
0 -0.29206216 -0.0033570339
0 -0.35584584 -0.0067140702
0 -0.40620136 -0.0067140702
0 -0.47334209 -0.0067140702
0 -0.55055392 -0.0067140702
0 -0.61098057 -0.0067140702
0 -0.70162052 -0.010071106
My implementation is
fileID = fopen('testdata.dat');
formatSpec = '%d %f %f';
clear C
k = 0;
N = 11;%Block size
while ~feof(fileID)
k = k+1;
C(k,1:3) = textscan(fileID,formatSpec,N,'HeaderLines', 8);%'CommentStyle','##',
end
fclose(fileID)
Please correct my code. Thanks in advance.
  2 Comments
Raj Raj
Raj Raj on 25 Oct 2014
p.s.
I tried the code given in Here However, I am getting error with my data file as
Index exceeds matrix dimensions.
at
ncol = numel( regexp( buf{1}, ' +', 'split' ) );
Appreciate your comments.
per isakson
per isakson on 27 Oct 2014
Edited: per isakson on 27 Oct 2014
That function read one trailing block of data after a file header, the last word of which was "END".

Sign in to comment.

Sign in to answer this question.

Accepted Answer

per isakson
per isakson on 27 Oct 2014
Edited: per isakson on 27 Oct 2014
Modified version of your code
fid = fopen('RajRaj.txt');
formatSpec = '%d%f%f';
clear C
k = 0;
N = 11; %Block size
while ~feof( fid )
k = k+1;
C( k, 1:3 ) = textscan( fid, formatSpec, N, 'HeaderLines', 8 );
end
fclose( fid );
check the output
>> whos C
Name Size Bytes Class Attributes
C 2x3 1112 cell
"Please correct my code" &nbsp I didn't try your code and isn't sure why it doesn't work. It's tricky to count header-lines. An ending new-line matters.
&nbsp
Here is another function that uses &nbsp fileread &nbsp and &nbsp regexp
>> g = read_blocks_of_numerical_data( 'RajRaj.txt', 150 );
>> g{:}
ans =
0 -0.1947 -0.0067
0 -0.1981 -0.0034
...
ans =
0 -0.1679 -0.0034
0 -0.2081 -0.0067
...
Where both the function and the sample text file are attached. The magical number &nbsp 150 &nbsp is the minimum number of bytes in a block of numerical data. The entire file must fit in memory together with the result.
&nbsp
And one more function
>> M = RajRaj()
M(:,:,1) =
0 -0.1947 -0.0067
0 -0.1981 -0.0034
...
M(:,:,2) =
0 -0.1679 -0.0034
0 -0.2081 -0.0067
...
where
function M = RajRaj()
xpr = '(?<=kN)[ 0-9\.\+\-e\r\n]++(?=(Data Header:)|(\s*$))';
str = fileread( 'RajRaj.txt' );
cac = regexp( str, xpr, 'match' );
M = nan( 11, 3, length( cac ) );
for jj = 1 : length( cac )
M( :, :, jj ) = str2num( cac{jj} );
end
end
&nbsp
I believe there is a reason why Matlab doesn't include a good functions to read a file with many blocks of numerical data.
  1 Comment
Raj Raj
Raj Raj on 27 Oct 2014
Hi Thanks for attention and time to respond. Here comes an issue after dealing with large data file.
My data is in chunks of 11 lines. However, by some reason machine wrote down a data block whose size is 8. However, the "textscan" with chunks data option also tries to match format of data. Hence, I am saved and read the second chunk of data whose size is 8. Here comes the trouble in reading the rest of the data.
The problem is in dealing with the 'Headerlines' option in textscan. Is there any way to tell textscan to go one line backwards ?
Appreciate your inputs. Thanks in advance.

Sign in to comment.

More Answers (2)

Geoff Hayes
Geoff Hayes on 25 Oct 2014
Raj - presumably, each block of 8 lines that you wish to ignore is of the format
<empty line>
Data Header:
Time: 100.16064 Sec 10/8/2012 3:32:17 PM
Data Acquisition: T
Station Name: default
Test File Name:
Axial Axial Axial
segments mm kN
So there is a blank/empty line followed by seven lines with text. Each of these blocks is then followed by 11 lines that you wish to keep.
I stepped through code (using the debugger) and noticed that the first iteration of your loop worked fine - three columns of 11x1 was read successfully. However, subsequent reads failed with either nothing read, or only 4x1 columns, etc. I repeated the above and once the first block was successfully read in, I typed the following in the Command Window to see the contents of the current line
K>> fgetl(fileID)
ans =
''
which makes sense if the next line is blank. I then repeated it twice more
K>> fgetl(fileID)
ans =
''
K>> fgetl(fileID)
ans =
Data Header:
So there seemed (for some reason) to be two blank lines between each block, rather than the expected one. That suggests a problem with the format specification string which you have initialized to
formatSpec = '%d %f %f';
If I were doing this, reading in data line by line, I would add the newline "character" to the end of the format as
formatSpec = '%d %f %f\n';
And that seems to be what was missing. Using your above data as an example, both blocks were read in successfully.
Just change your formatSpec to the above, and try again.
  1 Comment
Raj Raj
Raj Raj on 27 Oct 2014
Hi Thanks alot for your time. I tried but not able to reproduce the result as your mentioned. Instead I used following code to get the result:
fileID = fopen('testdata.dat');
formatSpec = '%f %f %f';
clear C
k = 0;N = 11;%Block size
while ~feof(fileID)
k = k+1;
if k==1,
C(k,1:3) = textscan(fileID,formatSpec,N,'HeaderLines', 8);%'CommentStyle','##',
else,
C(k,1:3) = textscan(fileID,formatSpec,N,'HeaderLines', 9);%'CommentStyle','##',
end
end
fclose(fileID)

Sign in to comment.

Gabriel Felix
Gabriel Felix on 24 May 2020
I had to use \n at the end of each line. Without it I couldn't make textscan() work properly, even thoug the "HeaderLines" was configured according to the text file lines. This was the only solution I found after struggling with the code for an intire day.
This was the text:
!
!
! alfa (graus) = 5.0
!
! Id. x/s z/s alfai cl c*cl/cmed cdi cmc/4
! (graus)
1 .246 .050 -1.209 .255 .332 .00538 .0170
2 .292 .150 -1.098 .259 .319 .00496 .0545
3 .339 .250 -.925 .254 .297 .00410 .0944
4 .385 .350 -.741 .243 .268 .00315 .1341
5 .432 .450 -.561 .227 .235 .00223 .1714
6 .479 .550 -.393 .206 .199 .00141 .2034
7 .525 .650 -.238 .181 .163 .00075 .2266
8 .572 .750 -.101 .152 .126 .00027 .2362
9 .619 .850 .014 .116 .089 -.00003 .2236
10 .659 .938 .103 .074 .052 -.00013 .1693
!
! CL asa = .208
! CDi asa = .00258
! e (%) = 88.9
! CMc/4 asa = .1339
My code:
%! alfa (graus) = 5.0
P = textscan(fid,'! alfa (graus) = %f','Delimiter',' ','MultipleDelimsAsOne',true,'headerLines',2,'CollectOutput',1);
alpha(1) = P{1};
%! CL asa = .208
P = textscan(fid,'! CL asa = %f\n','Delimiter',' ','MultipleDelimsAsOne',true,'CollectOutput',1,'headerLines',4+n);
CL(1) = P{1};
%! CDi asa = .00258
P = textscan(fid,'! CDi asa = %f\n','Delimiter',' ','MultipleDelimsAsOne',true,'CollectOutput',1,'headerlines',0);
CDi(1) = P{1};
%! CMc/4 asa = .1339
P = textscan(fid,'! CMc/4 asa = %f','Delimiter',' ','MultipleDelimsAsOne',true,'CollectOutput',1,'HeaderLines',2);
Cmc4(1) = P{1};

Categories

Find more on Logical in Help Center and File Exchange

Tags

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!