Rank: 4 based on 3413 downloads (last 30 days) and 52 files submitted
photo

John D'Errico

E-mail
Company/University
Retired

Personal Profile:

Mainly retired from Eastman Kodak. (Of course, Kodak itself is now semi-retired. I don't think I had any influence in that.) I still write MATLAB code as I find something interesting, but I DON'T answer your questions, and I do NOT do homework. Your homework is YOUR problem, not mine. Do NOT e-mail me with your homework problems or student projects. When I'm not doing something with MATLAB, you might find me playing bridge.

Professional Interests:
Bridge, MATLAB, numerical analysis, mathematical modeling

 

Watch this Author's files

 

Files Posted by John View all
Updated   File Tags Downloads
(last 30 days)
Comments Rating
23 May 2014 Screenshot A suite of minimal bounding objects Suite of tools to compute minimal bounding circles, rectangles, triangles, spheres, incircles, etc. Author: John D'Errico miminum, bound, bounding, circle, rectangle, triangle 91 16
  • 4.66667
4.7 | 6 ratings
29 Apr 2014 polyfitn Polynomial modeling in 1 or n dimensions Author: John D'Errico polyfit, modeling, regression, linear regression, approximation, function 267 43
  • 4.95833
5.0 | 24 ratings
29 Apr 2014 Screenshot SLM - Shape Language Modeling Least squares spline modeling using shape primitives Author: John D'Errico spline, splines, cubic, hermite, breaks, knots 160 101
  • 5.0
5.0 | 52 ratings
29 Jan 2014 Moving window standard deviation A (fast) windowed std on a time series Author: John D'Errico standard deviation, std, window, filter, movingavg 99 16
  • 4.72727
4.7 | 11 ratings
30 Oct 2013 HPF - a big decimal class High precision floating point arithmetic, a new class written in MATLAB Author: John D'Errico hpf, multiple precision, big decimal, floating point, arithmetic, mathematics 52 17
  • 5.0
5.0 | 8 ratings
Comments and Ratings by John View all
Updated File Comments Rating
23 Aug 2014 Remove outliers Turns outliers from a vector or matrix to NaN based on modified Thompson Tau method Author: M Sohrabinia

What Jered is trying to point is indeed a serious flaw in this code.

Consider these two examples. In the first case, I'll create an outlier that has large absolute value.

x = rand(10,1) + 10;
x(3) = 20;
y = outliers(x)
y =
10.439
10.382
NaN
10.795
10.187
10.49
10.446
10.646
10.709
10.755

outliers finds the element that is inconsistent with the data, and replaces it with nan. No problem there.

However, suppose we create a point that is just as much of an outlier? (In fact, it is more significantly an outlier by some measures, if we compare it to the mean of the remainder of the data.)

x(3) = 0;
y = outliers(x)
y =
10.439
10.382
0
10.795
10.187
10.49
10.446
10.646
10.709
10.755

See that outliers fails to identify that point.

The problem is that this code does indeed look at the simple magnitude of the point. Points with large magnitude are flagged as an outlier.

Another example might be appropriate. I'll create a nice smooth curve here.

t = linspace(0,2*pi,21)';
x = sin(t);
x(11)
ans =
1.2246e-16

So the 11th point is zero. (essentially.) I'll change it to 1, a value that is clearly not on the nice smooth curve.

x(11) = 1;
y = outliers(x)
y =
0
0.30902
0.58779
0.80902
0.95106
1
0.95106
0.80902
0.58779
0.30902
1
-0.30902
-0.58779
-0.80902
-0.95106
-1
-0.95106
-0.80902
-0.58779
-0.30902
-2.4493e-16

Oops, outliers fails to find a point that is obviously not on the curve. I'll make the outlier more obvious this next time. Clearly x here will be always positive, but I'll make one element fairly clearly an outlier, thus completely inconsistent with the remainder of the curve.

x = sin(linspace(0,2*pi,10)')+2;
x(5) = -2
x =
2
2.6428
2.9848
2.866
-2
1.658
1.134
1.0152
1.3572
2

y = outliers(x)
y =
2
2.6428
2.9848
2.866
-2
1.658
1.134
1.0152
1.3572
2

Again, outliers fails to find that point.

Jered has pointed out that this tool assumes the data is centered around zero, then it looks for points that have large absolute magnitude, and discards those points.

So while I like SOME ASPECTS about the documentation in this tool, and I like the way the code was written, the tool simply has a major flaw. I've not looked at the references, so I do not know if the basic underlying algorithm is just poor, or if the implementation is poor. This code fails to find outliers in your data, UNLESS those outliers are of a very specific class, thus large in magnitude compared to the rest of the data. The code fails to use any information about whether the curve is smooth or noisy, it merely looks for points that are large in absolute value compared to the rest.

There are other problems with this tool. The author has put a blank line into the code after the fist line. So despite the voluminous set of help written, when you use help, you get essentially no help at all.

>> help outliers
outliers function: remove outliers based on Thompson Tau:

>>

Help looks for a CONTIGUOUS block of comments, and dumps that block out to the command line. A blank line in the middle cuts help off at the knees.

I wanted to give this code 3 stars because the author made some effort despite the MAJOR flaw with the algorithm, but the lack of usable help reduces my rating to 2 stars.

To the author, don't tell me I don't understand the code.

To any potential users, I would suggest you do as Jered said - be very careful about using this code. Code that finds only a very restricted class of outlier is not good at all, unless the only outliers you will ever see are those it can find.

09 Aug 2014 inpaint_nans Interpolates (& extrapolates) NaN elements in a 2d array. Author: John D'Errico

Citing files on the FEX has come up often, enough so that we had a discussion about it, and came up with several schemes that seem viable. Find them at this link:

http://blogs.mathworks.com/community/2010/12/13/citing-file-exchange-submissions/

All you need do is modify the citation style you prefer to fit inpaint_nans, instead of gridfit.

09 Aug 2014 ColorBrewer: Attractive and Distinctive Colormaps The complete palette of ColorBrewer colormaps. Simple selection by scheme name and map length. Author: Stephen Cobeldick

A very nice submission to upgrade your figures for good viewing by all viewers.

Excellent help and examples for use. I found the colorbrewer.org web site interesting too.

07 Aug 2014 ismywiferight.zip This code gives an out put 'yes' regardless of the input. Author: Vineet

Oh, come on, give me a break.

05 Aug 2014 RMSEARCH Randomly multiply started optimizations for global problems Author: John D'Errico

This is a basic of MATLAB. You need to do the same thing for ANY optimizer, for any numerical integration method, etc.

What happens when you try passing in fun as an argument? If you call it like that:

[xfinal,ffinal,exitflag,xstart] = rmsearch(fun,'fmincon',...);

Look at it from the point of the MATLAB interpreter. MATLAB sees fun as a function, that it tries to execute immediately. It would then try to pass the result of that function call to rmsearch.

But that is NOT what you want to do. You want to pass in the function itself into rmsearch. You do this most easily via function handles, the things with an @ symbol in front of a function name. They also let you create a function on the fly. (Read about function handles in the help.)

So in this case, what you had to do was this:

[xfinal,ffinal,exitflag,xstart] = rmsearch(@fun,'fmincon',...);

Comments and Ratings on John's Files View all
Updated File Comment by Comments Rating
18 Aug 2014 HPF - a big decimal class High precision floating point arithmetic, a new class written in MATLAB Author: John D'Errico Shaoyang

12 Aug 2014 fminsearchbnd, fminsearchcon Bound constrained optimization using fminsearch Author: John D'Errico Sani

12 Aug 2014 polyfitn Polynomial modeling in 1 or n dimensions Author: John D'Errico JinSun

Incredible. Thank you for your contribution!

11 Aug 2014 distance2curve Find the closest point on a (n-dimensional) curve to any given point or set of points Author: John D'Errico Genevieve

09 Aug 2014 inpaint_nans Interpolates (& extrapolates) NaN elements in a 2d array. Author: John D'Errico D'Errico, John

Citing files on the FEX has come up often, enough so that we had a discussion about it, and came up with several schemes that seem viable. Find them at this link:

http://blogs.mathworks.com/community/2010/12/13/citing-file-exchange-submissions/

All you need do is modify the citation style you prefer to fit inpaint_nans, instead of gridfit.

Contact us