Thread Subject:

MATLAB Contest Starts Tomorrow - November 6th

Hello Everyone,


The latest MATLAB Programming Contest will run from November 6th to
13th. You can access the contest from the following page. Topic and
rules will be posted at the site tomorrow morning:


 <http://www.mathworks.com/contest>


These contests are a great way to learn new programming tricks, as
well as show off the ones you have. If you are curious about past
MATLAB Contests, follow the link above. The links to the past
contests are listed under "Contest Description" on the main page.


Please use this thread to talk about the contest, strategies or ask
related questions. If you have an "administrative" type question that
you don't feel applies to anyone else, e-mail us at
contest@mathworks.com.


The contest will start tomorrow at 9AM EST so make sure you set aside
some time during the week to take part. Good luck!


Min Poh
The MATLAB Central Team
The MathWorks, Inc.

May I ask what version of matlab will be used in the contest? If it is
6.5, then I can only be a spectator.... Sigh.....

Min Poh <mpoh@mathworks.com> wrote:

> The latest MATLAB Programming Contest will run from November 6th to
> 13th. You can access the contest from the following page. Topic and
> rules will be posted at the site tomorrow morning:

[ snip ]

Hiu Chung Law wrote:
> May I ask what version of matlab will be used in the contest? If it
is
> 6.5, then I can only be a spectator.... Sigh.....
>


The last two contests I've done with matlab 5.2. Except some small
problems (especially trying some submissions) it was no problem.


So you can do more than watching if you don't have the right version.


And, ..., you can also do submissions even without matlab. A big
problem then is that you can't test anything.


Stijn

The contest machinery will be using 6.5, but as Stijn points out, you
shouldn't let this deter you. The code distributed in the zip file
(which will be available when the contest begins tomorrow morning)
should work fine in previous versions, although I haven't tested it.

Once you have debugged your entry, then you submit it via the web
interface, and we take it from there. Technically, you don't need
MATLAB at all in order to play the contest. You can type your code
directly into the web interface and submit it (I wouldn't recommend
this, since the error messages aren't always useful across the web
interface). Good Luck.

Mike

Let the games begin!!

I think it should edit the runcontest.m
line 30 may be changed to "pA(a<1) = [];"


"Min Poh" <mpoh@mathworks.com> wrote in message
news:eeb4a65.-1@WebX.raydaftYaTP...
> Hello Everyone,
>
>
> The latest MATLAB Programming Contest will run from November 6th to
> 13th. You can access the contest from the following page. Topic and
> rules will be posted at the site tomorrow morning:
>
>
> <http://www.mathworks.com/contest>
>
>
> These contests are a great way to learn new programming tricks, as
> well as show off the ones you have. If you are curious about past
> MATLAB Contests, follow the link above. The links to the past
> contests are listed under "Contest Description" on the main page.
>
>
> Please use this thread to talk about the contest, strategies or ask
> related questions. If you have an "administrative" type question that
> you don't feel applies to anyone else, e-mail us at
> contest@mathworks.com.
>
>
> The contest will start tomorrow at 9AM EST so make sure you set aside
> some time during the week to take part. Good luck!
>
>
> Min Poh
> The MATLAB Central Team
> The MathWorks, Inc.

runcontest has some errors :
if you let it draw everything is going too fast (a pause is
forgotten?)
the drawing works, but not as wanted.
   pA(a<0) = [];
should be
  pA(a<=0) = [];


Stijn

I had left out the pause for debugging purposes. When you hit an
error, the current solution is plotted prior to the error checking.
If you want to look at the solution to each chain in the testsuite,
just add a pause command into the runcontest code.


Thanks for the heads up on the visualization code...I am resubmitting
the zip file in the next few minutes.


Stijn Helsen wrote:
>
>
> runcontest has some errors :
> if you let it draw everything is going too fast (a pause is
> forgotten?)
> the drawing works, but not as wanted.
> pA(a<0) = [];
> should be
> pA(a<=0) = [];
>
> Stijn
>

E. Brian Welch's entry "Try Again" found a hole in our validation
tests. This isn't the first time he's gotten one by us. We knew
something strange was happening when entries started coming in with
1/10th the score! We've plugged it up and re-scored those entries.


<http://www-sandbox.mathworks.com/contest/protein.cgi/view_submissi
on.html?id=7917>


<http://www-sandbox.mathworks.com/contest/mastermind.cgi/cheating.h
tml>

Sorry, those URLs should be:


 <http://www.mathworks.com/contest/protein.cgi/view_submission.html?id=7>
917


 <http://www.mathworks.com/contest/mastermind.cgi/cheating.html>

The author of the best entry before 3PM EST on Friday will win a
MathWorks T-shirt and a MathWorks stress-relieving squishy ball.
You'll also be added to the list of MATLAB Contest winners.


Check out the list of winners from the last contest:


 <http://www.mathworks.com/contest/molecule.cgi/analysis.html>


Good luck!

To get a sense of absolute performance, I've calculated a _lower_
_bound_ on the average energy for the test suite: 1777.


Summary: for each test case, place the N hydrophobic amino acids on
the unit grid in a minimum energy configuration (approximately a
disc), regardless of the given structure of the protein. That gives a
lower bound on the energy for that protein. No algorithm that also
considers protein structure can do better.


Of course, it's probable even that average energy level is
unachievable. But, a lower average energy is mathematically
impossible (barring a bad assumption in the calculation of the
minimum energy state).


Quick facts: The current leader, rlesnake_53, scores an average
energy of 1940 in the test suite. 17 of 106 tests are less than 1
unit energy from the minimum configuration, 11 of which were probably
"non-trivial" (having two or more hydrophobic amino acids).


Cory

According to the rules, the scoring algorithm is:


score = k1 * result + k2* exp (k3*runtime)


Seeing as the score, result and runtime are published I thought I'd
use FMINSEARCH to determine the constants. I minimise a function that
returns the mean square error between a fit to [result runtime] and
the score.


k1 = 0.99999999956405
k2 = 0.14285724741638
k3 = 0.04999941039568


Interesting?

Ed wrote:
> score = k1 * result + k2* exp (k3*runtime)
...
> k1 = 0.99999999956405
> k2 = 0.14285724741638
> k3 = 0.04999941039568


Nice reverse-engineering! These are correct to several digits.


Determining this formula is one of the trickiest contest-design
issues. Ideally, we're looking for an interplay between result and
runtime. We want to encourage both smart code and fast code, placing
a bit more importance on result, but enough emphasis on time to keep
the queue moving. We try to work out the constants by running an
internal contest, but we are often surprised by what happens when we
open the challenge up to the whole world.


This is the first time we've tried an exponential dependence on
runtime. The runtime penalty rises quickly as you near the maximum
time allowed. We wanted to keep entries from "playing the edge" of
the timeout, as it tends to encourage tweak bombing with many entries
that fail anyway.


For a definition of tweak bombing, see this analysis from a previous
contest:
 <http://www.mathworks.com/contest/surveyor.cgi/evolution.html>


What do you think of the exponential approach? It seems to be
working well so far. Can you think of any other ways to tackle this
problem?


Keep that feedback coming.


Sincerely,
Matthew J. Simoneau
The MathWorks, Inc.

Hello,


although (or perhaps because) I'm a 'tweaker',
I like to see the entry with the best result
as the leader. This morning, some of Stijn's
slight improvements didn't get on top, just
because of 10s of run time. Please as much
emphasis on result as possible. What about
sorting for result, then run time?


Heinrich


Matthew Simoneau wrote:
>
> What do you think of the exponential approach? It seems to be
> working well so far. Can you think of any other ways to tackle this
> problem?
>

>
> <http://www.mathworks.com/contest>
>
> These contests are a great way to learn new programming tricks, as
> well as show off the ones you have. If you are curious about past


While it is true that these contests are fun to watch, I doubt that
many "learn new programming tricks" from them. It is next to
impossible for outside observers (i.e., those who watch but don't
submit) to track changes in submitted code as the contest proceeds.
Since the submitted code does not use any or many comments and since
the code often uses cryptic single-letter variables, only those folks
who have the time and desire to win the contest take the time to
study what's been done by others and improve it.


I also doubt that the contests give folks the chance to "show off"
programming tricks either. Once again the code is so cryptic and
poorly documented that learning anything from it requires a
substantial investment in time and effort.


To me, the number one thing I learn from these contests is the
fundamental need to write good, readable, well-commented code.
Cryptic code may run a little faster, but overall productivity falls
to near zero as soon as someone says, "Can you modify this code
written by someone else to do _______?"

Hi, I posted essentially the same constants yesterday on the
newsgroup, but didn't make it into the right thread I guess. Anyway,
I ask the same question: given that they are easy to reverse
engineer from the data, why are they not published?


Here's a script to plot the constant score contours for the results
and runtimes of interest:
%%%%%%%%%%%%%%%%%%%%
[result,runtime] = meshgrid(1700:10:2500,0:180); %CorySharp lower
bound = 1777
k = [1 1/7 1/20];
score = k(1).*result+k(2).*exp(k(3)*runtime);
contour(1700:10:2500,0:180,score,20);
xlabel('result');
ylabel('run time');
%%%%%%%%%%%%%%%%%%%%%%


I believe it demonstrates the benefits of exponential method... for
long runtimes it is much easier to improve ones score by reducing the
runtime than by improving the result. By Heinrich's measure, either
would be attractive, making for large average runtimes.
Ken


 
--------------------
Heinrich Acker wrote:
>
>
> Hello,
>
> although (or perhaps because) I'm a 'tweaker',
> I like to see the entry with the best result
> as the leader. This morning, some of Stijn's
> slight improvements didn't get on top, just
> because of 10s of run time. Please as much
> emphasis on result as possible. What about
> sorting for result, then run time?
>
> Heinrich
>
> Matthew Simoneau wrote:
>>
>> What do you think of the exponential approach? It seems to be
>> working well so far. Can you think of any other ways to tackle
this
>> problem?
>>
>

Hello,


Yesterday there was a problem with my connection. Now (and on a "bad
moment") I get to the internet (and this page), but the contest page
can't be read ?????


I hope this problem will be solved soon (after 8 minutes the early
bird ... is over).


Stijn

"Heinrich Acker" <Heinrich.Acker@web.de> writes:

>although (or perhaps because) I'm a 'tweaker',
>I like to see the entry with the best result
>as the leader. This morning, some of Stijn's
>slight improvements didn't get on top, just
>because of 10s of run time. Please as much
>emphasis on result as possible. What about
>sorting for result, then run time?

Another tweaker's opinion: If you want to make the score count while still
avoiding the time limit, how about having no time penalty for the first two
minutes (say). Then, after that, you could impose a penalty, either linear
or exponential.

In that way, tweakers would have nothing to gain from shaving seconds of
submissions which are relatively fast already.

BTW, what's up with the Mathworks web server right now? I can't seem to
reach it.

Nicke
--
                    "A witty saying proves nothing."
                              - Voltaire (1694-1778)

"Stijn Helsen" <SHelsen@compuserve.com> writes:

>Yesterday there was a problem with my connection. Now (and on a "bad
>moment") I get to the internet (and this page), but the contest page
>can't be read ?????


>I hope this problem will be solved soon (after 8 minutes the early
>bird ... is over).

Indeed, it smells like a denial of service attack. Especially now
since the server is reachable again after the early bird deadline.

I noticed the same thing...

Nicke

--
                    "A witty saying proves nothing."
                              - Voltaire (1694-1778)

Our web server went down for at least 15 minutes right before the 3PM
deadline for the Early-bird Contest. We're very sorry about that. We
will still award the best entry before 3PM EST. We will also award the
best entry submitted before 4PM EST.

Mike

Ju win's our Early-bird Contest with entry h12_1. Congratulations to
our first contest winner! Ju, please send your contact information
to contest@mathworks.com so we can mail you your prizes. This entry
held the top spot for a remarkable 70 minutes. It is a tweak of
sragner's groundbreaking entry opttest5, which beat the previous best
score by more than 6 points.


Because of the server downtime, we're awarding a second early-bird
prize at 4PM.

Yuval Cohen's "rice_2" edged out the competition in the second round
of our Early-bird Contest. He and Ju will each receive a MATLAB
T-shirt and a stress-relieving MATLAB squishy ball. In addition, they
will be added to our illustrious list of contest winners.

We have posted a Mid-Contest Analysis. It illustrates the evolution
of the contest so far and picks apart one of the top entries.


 <http://www.mathworks.com/contest/protein.cgi/midcontest.html>


We've also posted a story of how one persons's innovation was
recognized by another and incorporated into a first-place entry.


 <http://www.mathworks.com/contest/protein.cgi/story.html>


See you in the queue.

Ed wrote:
>
>
> According to the rules, the scoring algorithm is:
>
> score = k1 * result + k2* exp (k3*runtime)
>
> Seeing as the score, result and runtime are published I thought I'd
> use FMINSEARCH to determine the constants. I minimise a function
that
> returns the mean square error between a fit to [result runtime] and
> the score.
>
> k1 = 0.99999999956405
> k2 = 0.14285724741638
> k3 = 0.04999941039568
>
> Interesting?
>


I think it is more like


k1 = 1
k2 = exp(-1.95)
k3 = 0.05


I cannot think of any reason for choosing 0.99999999956405 instead of
1.


So


score = result + exp(-1.95 + 0.05*runtime)

Close, I think,
but k2 = 0.14285724741638 = 1/7


Kevin Spiteri wrote:
>
>
> Ed wrote:
>>
>>
>> According to the rules, the scoring algorithm is:
>>
>> score = k1 * result + k2* exp (k3*runtime)
>>
>> Seeing as the score, result and runtime are published I thought
I'd
>> use FMINSEARCH to determine the constants. I minimise a function
> that
>> returns the mean square error between a fit to [result runtime]
and
>> the score.
>>
>> k1 = 0.99999999956405
>> k2 = 0.14285724741638
>> k3 = 0.04999941039568
>>
>> Interesting?
>>
>
> I think it is more like
>
> k1 = 1
> k2 = exp(-1.95)
> k3 = 0.05
>
> I cannot think of any reason for choosing 0.99999999956405 instead
of
> 1.
>
> So
>
> score = result + exp(-1.95 + 0.05*runtime)
>

In the contest I read some comments about the long codes, and parts
which are not or almost not used.
Some of the code (maybe a big part) is not or almoset not necessary.
But if a part is used only once, where the improvement of the score
is (more then) enough to compensate for the loss in CPU-time, that
part contributes to the code.
And also in the method list, there is the "spiral", which also came
from me.


Stijn

In the contest I read some comments about the long codes, and parts
which are not or almost not used.
Some of the code (maybe a big part) is not or almoset not necessary.
But if a part is used only once, where the improvement of the score
is (more then) enough to compensate for the loss in CPU-time, that
part contributes to the code.
And also in the method list, there is the "spiral", which also came
from me.


Stijn

Stijn Helsen wrote:
>
>
> ana004 just came over joshua TT2 with 0.5 s less. That means more
> tests -- less time ?
> (is s==1 faster then s<2?)


Maybe, but hardly as much as our latest "tests" suggests. I believe
that my "improvement" is due to variations in the scoring system,
mostly.


/Anders

The timing differences seen between ana004
and ana004-copy have been important in previous
contests also. I have tried copies of a
mastermind solver in that contest.


It seems that the run time is a bit a matter of
luck due to the many tasks Windows runs.


To prevent this - and to support work between contests - I would like
to have something like
the old flops count back. Optimizations in
general are more productive if the measure used
is exactly reproductive. One run is enough, then.


Heinrich

I guess the right word is reproductible ...


Heinrich Acker wrote:
>


> general are more productive if the measure used
> is exactly reproductive. One run is enough, then.

(accidently posted in a new thread)


Dear contestants,


since I can't find through the large code
labyrinth of the current contributions any
more, I decided to publish my ideas:


1. a) According to Message 11, calculate the
   lower bound of energy. Improve the method by
   taking into account that single zeroes
   bracketed by ones can't be pushed out of the
   disc (results in a more realistic lower
   bound).
   b) Sort the solver algorithms for their
   contribution to the final solution according
   to the mid-contest analysis.
   c) replace each


   if ec<bc
      bc=ec;
      c= ... ;
   end


   with


   if ec<bc
      c= ... ;
      if ec<bc_opt
         return
      end
      bc=ec;
   end


   d) I suggest 'btq' as the basis because of
   the best result so far.


2. 'energy' is very time-consuming. Together
   with 'energy1', it uses about 40% of the
   run time of ana004 or the like.
   Perhaps an approximate, but faster
   'energy' does the job, too:
   For each acid of the chain, 6 (7 at the
   ends) neighbour places are zero, one, or
   nothing, depending on folding. Sum up the
   number of non-one neighbours for each one
   to get a measure of the density of the
   pack. The higher the density, the smaller
   the sum.


Regards,


Heinrich

As I write these lines, the first five
entries in the list have the same result.
They are separated in score by the run time.
So even 100ms of run time are not negligible
if you want to win. That's why the observed
speed differences are a bit disturbing in the
contest.


Cory Sharp wrote:
>
> From observing Ken Crounse's contour, improvements in running time
> below 60s gives small to negligible improvements in score.

Actually, I don't think 100ms matters that much. These contests are
usually won by making a real improvement and submitting it in the
last 15 minutes of the contest when the queue is so long it won't
show up in first place until after the deadline.


Theoretically someone could win the contest by just submitting
duplicate copies of entries waiting in the cue before the deadline
but (a) Nobody would do that and (b) If someone did, the
post-analysis would still credit the real winner.


Another "problem" is that of nondeterministic code. This can have a
far greater effect than 100ms would have. See for instance my entry
"Iron Sulphide 3" which scored about a point better than the then #1.


/Per


Heinrich Acker wrote:
> So even 100ms of run time are not negligible
> if you want to win. That's why the observed
> speed differences are a bit disturbing in the
> contest.

> Actually, I don't think 100ms matters that much. These contests are
> usually won by making a real improvement and submitting it in the
> last 15 minutes of the contest when the queue is so long it won't
> show up in first place until after the deadline.
>
> Theoretically someone could win the contest by just submitting
> duplicate copies of entries waiting in the cue before the deadline
> but (a) Nobody would do that and (b) If someone did, the
> post-analysis would still credit the real winner.
I wouldn't say that people don't do that. And it has happened (more
then once) that the "real winning entry" didn't win but a "tweaked
one" (at least for the "intermediate winners").
Stijn

Per Rutquist wrote:
>
>
> Actually, I don't think 100ms matters that much.
In this contest it was not 100ms spread but seconds.

Moved this from a seperate post in newgroup so it would show. Also
tried to fix table spacing.


To help guide the algorithm developers I ran some statistics on the
test cases being used (runs by names length... and sum...). These
expand upon the work done in q1, q2, t4 etc. by unnamed author. It
appears as if:


There are 105 cases
Length breakdown:


Note: Previous runs of Timecheck3 and MaxLength=100 showed max
length was 99 or 100 so I had upper bound.


Length Number
  > 100 ...... 0
91 to 100 ... 17
81 to 90 ..... 8
71 to 80 ..... 9
61 to 70 ..... 12
51 to 60 ...... 9
41 to 50 ..... 10
31 to 40 ...... 7
21 to 30 ..... 15
16 to 20 ...... 6
10 to 15 ...... 2
 8 or 9 ....... 2
 6 or 7 ....... 2
 5 ............ 1
 4 ............ 2
less than 4 ... 3
total 105


Also did a sum which checks how many short strings there are and can
give a rough percentage of the two types of atoms


Sum of elements Number
   > 70 ....... 2
61 to 70 ....... 0
51 to 60 ....... 5
41 to 50 ....... 21
31 to 40 ....... 16
21 to 30 ....... 18
11 to 20 ....... 22*
 6 to 10 ....... 6
    5 ....... 2
    4 ....... 1
    3 ....... 4
    2 ....... 2
    1 ....... 4
    0 ....... 2
total 105


Note: On sum for 11 to 20 made typo so actually ran the same almost
identical test twice once got 23 as answer, other time got 22 so
picked 22 so total came out the same.


I hope this can help people with their algorithms.


Russ

On Sat, 9 Nov 2002 05:48:19 -0500, "Heinrich Acker"
<Heinrich.Acker@web.de> wrote:

>The timing differences seen between ana004
>and ana004-copy have been important in previous
>contests also. I have tried copies of a
>mastermind solver in that contest.
>
>
>It seems that the run time is a bit a matter of
>luck due to the many tasks Windows runs.

Especially since the function cputime has a bug on the Windows
platform where it measures the wall clock time instead of the cpu
time.

  Lars

Lars Gregersen
www.rndee.dk
lg@rndee.dk

When does the contest finish ? I know wed, but I have no time...


regards

The contest will end at 5pm EST on Wednesday November 13, 2002.

Lucio wrote:

> When does the contest finish ? I know wed, but I have no time...
>
>
> regards
>

I realize that you are all far more advanced in the ways of matlab
programming. For this reason, I think that it is stupid that no-one
is commenting their entries. I don't mean nice line-by-line
comments. I'm just talking about general comments. Entries like
stupid1 which doesn't even describe the tweaks and changes aren't
helpful. It takes no effort to say -- "speed up changes only" or
"this is a ball and twine pattern" or "change to evaluation function"


Sorry about the bitchin' and moaning.


Dave

You are right. I am sorry about not commenting my entry. But a diff
of my entry and its predecessor should show pretty clearly what I
did. I just added a couple of quick methods that work better on some
configurations. I didn't tweak any of the current methods.


D Holmes wrote:
>
>
> I realize that you are all far more advanced in the ways of matlab
> programming. For this reason, I think that it is stupid that no-one
> is commenting their entries. I don't mean nice line-by-line
> comments. I'm just talking about general comments. Entries like
> stupid1 which doesn't even describe the tweaks and changes aren't
> helpful. It takes no effort to say -- "speed up changes only" or
> "this is a ball and twine pattern" or "change to evaluation
function"
>
> Sorry about the bitchin' and moaning.
>
> Dave
>

Are you shocked by such fast speed now? That is a hole in the
runcontest program. I used global variables to remember previous
results so that you can match the results and return. Now anyone
copied these codes, they just repeat previous results what ever they
are correct or not. So your score is depending on other's results not
yours! Contest team may need a patch for the hole ('clear global' at
the begining of runcontest).

Aaargh! So much for all that work tuning the code :( Although Yi is
not quite correct - I managed to break the 2180 barrier by using both
the results from my algorithm and the results from others :)
Unfortunately, I didn't think TMW considered it a hole (having
double-checked the fine print in the rules) , which is why I devised
that approach.


Oh well... Back to the drawing board!


Sorry for the gripe :)


Yi Cao wrote:
>
>
> Are you shocked by such fast speed now? That is a hole in the
> runcontest program. I used global variables to remember previous
> results so that you can match the results and return. Now anyone
> copied these codes, they just repeat previous results what ever they
> are correct or not. So your score is depending on other's results
not
> yours! Contest team may need a patch for the hole ('clear global' at
> the begining of runcontest).
>

Maybe 8pm EST wasn't the best time to submit that hole. The people
who could fix the bug will probably be asleep for another five or six
hours.


/Per


Yi Cao wrote:
> Are you shocked by such fast speed now? That is a hole in the
> runcontest program.

I don't like that most of the current winning entries (since
yesterday) don't work with the testsuite.

Roger Stuckey wrote:
>
>
> Aaargh! So much for all that work tuning the code :( Although Yi is
> not quite correct - I managed to break the 2180 barrier by using
both
> the results from my algorithm and the results from others :)


Yi is exactly correct. I was going nuts trying to figure out why my
"improvements" made the running time so much worse -- now it seems
that I was unknowingly returning the results that someone else had
stored. You (knowing what was going on) stored your results in
location with your name, so your function returned only what it
computed itself.


Yi's suggested solution (clear global for each new solver) will allow
programs to be rerun without modification, and those of us who were
spending our work tuning without knowing the hole will be able to
find out the true results of our functions. Your true result, Roger,
should be unchanged, although the CPU time will be higher.


>
> Yi Cao wrote:
>>
>>
>> Are you shocked by such fast speed now? That is a hole in the
>> runcontest program. I used global variables to remember previous
>> results so that you can match the results and return. Now anyone
>> copied these codes, they just repeat previous results what ever
they
>> are correct or not. So your score is depending on other's
results
> not
>> yours! Contest team may need a patch for the hole ('clear
global' at
>> the beginning of runcontest).
>>
>

It would be nice if the MATLAB contest team
could provide an explanation of 'the hole'
(after fixing it) that is clear not only
for the pros. I didn't get it yet.


1) How is it possible that all the leading
entries couldn't finish the test suit at
home?
2) How is it possible that an entry stores
information, but a tweak of that entry
does not benefit?


Heinrich

Heinrich Acker wrote:
>
>
> It would be nice if the MATLAB contest team
> could provide an explanation of 'the hole'
> (after fixing it) that is clear not only
> for the pros. I didn't get it yet.
>
> 1) How is it possible that all the leading
> entries couldn't finish the test suit at
> home?
> 2) How is it possible that an entry stores
> information, but a tweak of that entry
> does not benefit?
>
> Heinrich
>


1) I think you meant the code originally contributed by Lucio
Andrade. There was a bug there. Now it has been fixed by Yuval. See
 <http://www.mathworks.com/contest/protein.cgi/view_submission.html?id=9>
696
I guess why this bug can pass contest check but cannot run on our
testsuit is due the difference between these two testsuits.
2) The code, which identifing the hole, stores current results if the
global variable is empty, or just returns the previous results if the
global variable exists and the protein is matched. Therefore, it is
understandable why a tweak may have no effect on improvement because
it depends when do you submit the tweak, or exactly whom does your
tweak follows. You can devise this hole/code by a double-submision
with new global variable names as Roger did.


My code originally comes from the idea to check if the testsuit has
identical proteins. If so, why we need repeatedly calculating.
According to previous contest policy, they reboot PC before starting
very new entry. If so, this code is OK. Every entry needs creating
new global variables once. Surprisingly I found they changed the
policy and the code has made chaos. I am sorry for that.

That was me...I through a clear statement onto the server until I
could find a good way to clear the globals in between submissions.
All of the submissions submitted since 5pm yesterday are being
reprocessed. I believe Yi Cao's hack started around 6:30, so I went
a little further back for good measure.


I realize this is a huge inconvenience since the queue will be jammed
for a couple of hours, but it is the best we can do.


Mike

I understand the solution, with its problems. I find it a bit
curious that that problem occured.


Stijn

Yi Cao wrote:
> 2) The code, which identifing the hole, stores current results if
the
> global variable is empty, or just returns the previous results if
the
> global variable exists and the protein is matched.
...
> My code originally comes from the idea to check if the testsuit has
> identical proteins. If so, why we need repeatedly calculating.
> According to previous contest policy, they reboot PC before starting
> very new entry. If so, this code is OK. Every entry needs creating
> new global variables once. Surprisingly I found they changed the
> policy and the code has made chaos. I am sorry for that.


This is a nice summary of what happened. We don't reboot the PC
between every entry because that takes a few minutes and we want to
keep the queue moving. As far as we can tell, this GLOBAL hole has
been around for every contest up to now. What's most surprising is
that this hasn't come up before! Last contest we caught PERSISTENT,
this one it was GLOBAL. Each contest, we tighten the machinery a bit
more.


We're re-running all the entries that might have been affected, so
the queue will be jammed up for a couple hours. Sorry about the
delay.

Matthew Simoneau wrote:
>
>
> This is a nice summary of what happened. We don't reboot the PC
> between every entry because that takes a few minutes and we want to
> keep the queue moving. As far as we can tell, this GLOBAL hole has
> been around for every contest up to now. What's most surprising is
> that this hasn't come up before! Last contest we caught PERSISTENT,
> this one it was GLOBAL. Each contest, we tighten the machinery a
bit
> more.
>
> We're re-running all the entries that might have been affected, so
> the queue will be jammed up for a couple hours. Sorry about the
> delay.
Didn't you reboot the PC everytime last contests ? So how could this
global-problem come up last time ?


Stijn

Stijn Helsen wrote:
> Didn't you reboot the PC everytime last contests ?


Nope. We weren't rebooting the PC for every entry in earlier
contests either.

Matthew Simoneau wrote:
>
>
> Nope. We weren't rebooting the PC for every entry in earlier
> contests either.


I thought that that was said last time, and that that was a reason
why the queue was so long at the end of the contest. (I know that
most of the entries then, were close to the limit of 180 s, so that
startup time was not the only reason.) The fact that Yi Cao thought
so too, must be that it was said somewhere.


Stijn

Matthew Simoneau wrote:
>
>
> Stijn Helsen wrote:
>> Didn't you reboot the PC everytime last contests ?
>
> Nope. We weren't rebooting the PC for every entry in earlier
> contests either.
>
OK, the reboot condition is from my memory about the following
message originally from Mike posted for the last contest.


"Actually, the queue was fine...it was just bogged down. There were
close to 70
entries between 4 and 6 this evening, and the server only runs one at
a time. If an
entry times out, then there is a few more minutes to reboot the
machine in addition
to the 3 minutes to run most entries.


We realize this can be frustrating, but there is no way around it.
Thanks for
keeping on me about the queue, and if it looks like it is hung, don't
hesitate to
throw another message my way. Oh, and thanks for playing.


Mike"


Seems only when an entry times out, a reboot will happen. So, it was
my memory not correct.

Kevin Spiteri wrote:
>
> score = result + exp(-1.95 + 0.05*runtime)
>


The current top entry contains 1771 lines
of poorly documented M-code, making it al-
most impossible to grasp what's being changed
or added from one entry to the next.


I admit that all the submitted "spaghetti
codes" are very good at folding the proteins
from the contest suite, but I can't escape
the feeling that most GPRs (good programming
rules) are completely neglected. To put it
a bit sharper:


The contest discourages good programming.


Don't misunderstand me. I am enjoying the
contest and facinated by the zoology of
modules in the current code!


So what can be improved?
- Add the length of the p-code of an entry
  to its score. This will favour generic
  and compact solutions.
- Disclose code of new entries only when
  they are a few hours old. This helps
  participants to stick to their own ideas
  at least for a while before copying the
  tweaks from others into their codes.
- Let participants submit entries for
  testing purposes only. The scoring
  result will be sent back to the parti-
  cipant per e-mail. So one has time to
  work on an idea before it gets "open
  source". Every fifth contribution (or
  so) of a participant gets included into
  the official ranking, and its code
  disclosed.
- If the contest should encourage team
  work, why not set up teams that can
  register and share their private ranking
  for a while? One could combine the best
  codes from all private ranking lists into
  an official ranking every couple of
  hours.


Just ideas...


Last not least:
Please provide an entry download button in
addition to the edit button!


sTefan

> Please provide an entry download button in
> addition to the edit button!
>
> sTefan
>
This was asked last year (and maybe before too), but "they" don't do
it.


(It is funny now that you can open submissions which are marked as
"new", but with a score. (just remarking that not every data was
cleared))


Stijn

This is my first time participating in this contest and I have been
loving every minute of it!


I agree with Stefan's third point: to be able to submit for testing
purposes only and maybe with his first point: to consider code length.


Jose


Stefan Stoll wrote:
>
>
> Kevin Spiteri wrote:
>>
>> score = result + exp(-1.95 + 0.05*runtime)
>>
>
> The current top entry contains 1771 lines
> of poorly documented M-code, making it al-
> most impossible to grasp what's being changed
> or added from one entry to the next.
>
> I admit that all the submitted "spaghetti
> codes" are very good at folding the proteins
> from the contest suite, but I can't escape
> the feeling that most GPRs (good programming
> rules) are completely neglected. To put it
> a bit sharper:
>
> The contest discourages good programming.
>
> Don't misunderstand me. I am enjoying the
> contest and facinated by the zoology of
> modules in the current code!
>
> So what can be improved?
> - Add the length of the p-code of an entry
> to its score. This will favour generic
> and compact solutions.
> - Disclose code of new entries only when
> they are a few hours old. This helps
> participants to stick to their own ideas
> at least for a while before copying the
> tweaks from others into their codes.
> - Let participants submit entries for
> testing purposes only. The scoring
> result will be sent back to the parti-
> cipant per e-mail. So one has time to
> work on an idea before it gets "open
> source". Every fifth contribution (or
> so) of a participant gets included into
> the official ranking, and its code
> disclosed.
> - If the contest should encourage team
> work, why not set up teams that can
> register and share their private ranking
> for a while? One could combine the best
> codes from all private ranking lists into
> an official ranking every couple of
> hours.
>
> Just ideas...
>
> Last not least:
> Please provide an entry download button in
> addition to the edit button!
>
> sTefan
>

I strongly agree with the first two of Stefan's suggestions.


I don't think reply-per-email testing is a good idea because it would
discourage people from ever showing their ideas until the very end.


Teams could communicate just as easily using email as on the website,
so no modifications are necessary. (Just submit entries in the name
of your team.)


The idea of not displaying the code of new entries for some time is a
very good one.
For instance: All new entries are hidden for one hour before being
made public. If an entry reaches the top it is hidden for up to
twelve hours if it is not defeated by another entry. The author of an
entry that stays on top for twelve hours receives eternal glory, or a
T-shirt, or both.


I don't think this hiding of new entries would slow down the
evolution of code.
On the contrary, the chance of receiving eternal glory at any time
will encourage people to submit good ideas immediately.


(Personally I always like to sit on one good idea to use at the end
of the contest. This year I was going to keep the sum-of-square
distances idea to myself, write an algorithm that uses many random
guesses, combine it with the top entry at 4:45 and submit it 5
minutes before the deadline.


Then of course, when Brian Welch topped the list with "Better??" and
a comment saying that square distances worked as well as nonsquare
ones, I figured it would only be a matter of time before someone
thought of my idea, so I submitted my code. ("No, but faster!")


As a side note: This strategy has never worked for me before either.)


Yet another "scoring method" that I would like to see is to have all
entries rescored after the contest, using a very slightly different
problem set. Then a bonus prize could be awarded for "best code not
tweaked especially for the contest suite". At present, luck
contributes about five points to the score of the top entry. This
means that programs that do two points better on average but makes a
different number of randn()-calls score three points worse on the
server. (Programmers: A tip! Write your own randn-algorithm, or use
rand() instead.) The problem is that a lot of tweakers add bad code,
or remove good code at random, and this can improve the score just by
luck. (Cf "Iron Sulphide 3".)


(Lastly) Tweakers: You could use the rand() statement to identify
exactly which case you're working on. Don't use rand anywhere else in
your function, and the following will execute a piece of code on only
one of the entries. (The 50:th entry in the contest suite actually.)
 
r=rand;
if(r>0.18965374754716 & r<0.18965374754718)
  code...
end
 
This will enhance the tweaking, and make the contest even more
difficult for people who try to write good, general code.


Have fun, everyone!


/Per, who has to leave town for a meeting and will miss the rest of
the contest. :-(

I think this is not correct. Of course, the programming is poorly
documented and inefficient. But, incredibly, both the documentation
and efficiency increases as the contest goes along. What folks like
Lucio have done is an excellent example of proper coding techniques.


One problem of course is the extreme difficulty of the problem. PhD
theses could be written on the solution to it (or at least a good
masters thesis). Once again, I congratulate the Mathworks team on a
great problem selection.


Some of the other suggestions below may be good. But the system
doesn't appear to be broken to me.


Stefan Stoll wrote:
>
>
> Kevin Spiteri wrote:
>>
>> score = result + exp(-1.95 + 0.05*runtime)
>>
>
> The current top entry contains 1771 lines
> of poorly documented M-code, making it al-
> most impossible to grasp what's being changed
> or added from one entry to the next.
>
> I admit that all the submitted "spaghetti
> codes" are very good at folding the proteins
> from the contest suite, but I can't escape
> the feeling that most GPRs (good programming
> rules) are completely neglected. To put it
> a bit sharper:
>
> The contest discourages good programming.
>
> Don't misunderstand me. I am enjoying the
> contest and facinated by the zoology of
> modules in the current code!
>
> So what can be improved?
> - Add the length of the p-code of an entry
> to its score. This will favour generic
> and compact solutions.
> - Disclose code of new entries only when
> they are a few hours old. This helps
> participants to stick to their own ideas
> at least for a while before copying the
> tweaks from others into their codes.
> - Let participants submit entries for
> testing purposes only. The scoring
> result will be sent back to the parti-
> cipant per e-mail. So one has time to
> work on an idea before it gets "open
> source". Every fifth contribution (or
> so) of a participant gets included into
> the official ranking, and its code
> disclosed.
> - If the contest should encourage team
> work, why not set up teams that can
> register and share their private ranking
> for a while? One could combine the best
> codes from all private ranking lists into
> an official ranking every couple of
> hours.
>
> Just ideas...
>
> Last not least:
> Please provide an entry download button in
> addition to the edit button!
>
> sTefan
>

Stefan Stoll wrote:
>
>
> Kevin Spiteri wrote:
>>
>> score = result + exp(-1.95 + 0.05*runtime)
>>
>
> The current top entry contains 1771 lines
> of poorly documented M-code, making it al-
> most impossible to grasp what's being changed
> or added from one entry to the next.
>
> I admit that all the submitted "spaghetti
> codes" are very good at folding the proteins
> from the contest suite, but I can't escape
> the feeling that most GPRs (good programming
> rules) are completely neglected. To put it
> a bit sharper:
>
> The contest discourages good programming.
>
> Don't misunderstand me. I am enjoying the
> contest and facinated by the zoology of
> modules in the current code!
>


This restates in part what I said very early on in this thread, which
was:


>
> <http://www.mathworks.com/contest>
>
> These contests are a great way to learn new programming tricks, as
> well as show off the ones you have.


While it is true that these contests are fun to watch, I doubt that
many "learn new programming tricks" from them. It is next to
impossible for outside observers (i.e., those who watch but don't
submit) to track changes in submitted code as the contest proceeds.
Since the submitted code does not use any or many comments and since
the code often uses cryptic single-letter variables, only those folks
who have the time and desire to win the contest take the time to
study what's been done by others and improve it.


I also doubt that the contests give folks the chance to "show off"
programming tricks either. Once again the code is so cryptic and
poorly documented that learning anything from it requires a
substantial investment in time and effort.


To me, the number one thing I learn from these contests is the
fundamental need to write good, readable, well-commented code.
Cryptic code may run a little faster, but overall productivity falls
to near zero as soon as someone says, "Can you modify this code
written by someone else to do _______?"


*************************


I concur with Stefan's comments. I also like another one I read
somewhere: That execution time should not be penalized at all until
it reaches some significant percentage of the maximum time allotted.
That way well written algorithms that do not consume an excess amount
of time do not suffer from variations in the timing accuracy while
running the test suite.


In a past contest I tested this timing variability by simply
resubmitting a midcontest winning solution with no changes other than
adding a comment line to the code. After several such "faked"
submissions, "my" solutions were sometimes faster, sometimes slower
than the midcontest winner that I chose to test! If I could have done
this at the last minute and was lucky enough, then I could have been
the contest winner, despite the fact that I did no real work--I just
exploited a scoring system weakness.


As long as execution time and solution validity w.r.t. a given test
suite are the only measures that determine winning, these contests
will always gravitate to unreadable code that is a mixture of code
from a variety of submitters. If code readability is important (and I
believe it is), then it must contribute somehow to the scoring system.

Thanks Mr Keenan for your words, I can't Imagine why I am unemployed
now !... well, there is improvement possible in my part of code which
can be corrected now and may give less scoring, certainly speed could
also be improved but I see that the real winner will have to find
better sequences, at least in my opinion that's the one who should
win and there is time available for new routines...


things that can be done in my code
1. correct the error of the "Hidrophilic go away" folder
2. select more efficiently the direction to fold (in the HGA); when
we reach the half length of a sequence of zeros, right now I chose
one direction arbitrarily
3. Correct the end of the magic ball, I corrected the beginning and
it improved by 2 points, I expect something similar with the ends


Hey guys, I missed some of the names of the contributors, since we
all re-using the same code I ask every author involved to put your
name, at end Mathworks should give a prize to all of us.


Happy programming... cheers !

OK, I had some fun isolating a few cases from the scoring suite by
using CPUtime as a communication channel. (Refer to the "hairy" line
of submissions.) Basically, after determining that a input string
had a unique length among all strings in the suite, that string was
determined 5 bits at a time. Here are the results I got (which were
verified):
% length 10
a = [ 1 0 0 0 0 1 1 1 0 1]'
% length 34
a =[ 1 1 1 1 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0
1 1]'
% length 36
a =[ 0 0 1 1 0 1 0 1 0 0 0 0 0 1 1 1 0 0 1 0 0 0 1 1 0 1 1 1 1 0 0 0
0 1 1 0]'
also:
string length/ number in suite
% 31 0
% 32 0
% 33 0
% 34 1
% 35 0
% 36 1
% 37 0
% 38 0
% 39 3
% 40 ?
% 41 1
% 42 0
% 43 1
% 44 3
% 45 1
% 46 1
% 47 0
% 48 1
% 49 2


For the three strings that I believe I determined, I was able to come
up with better foldings than the front runners. But, somehow I
wasn't able to take advantage of that information, which I find a
little mystifying. Anyway, maybe this posting will be of use to
someone.


On the other hand I'm not sure this kind of behavior should be
encouraged. It would seem that the whole scoring suite could be
reverse engineered by a well motivated person. One solution to stop
this activity would be to have a non-discrete problem for our next
contest -- something I would like to see anyway -- Matlab is much
more fun when finding eigenvalues and solving differential equations.


Ken Crounse


Russell Newcomb wrote:
>
>
> Moved this from a separate post in newgroup so it would show. Also
> tried to fix table spacing.
>
> To help guide the algorithm developers I ran some statistics on the
> test cases being used (runs by names length... and sum...). These
> expand upon the work done in q1, q2, t4 etc. by unnamed author. It
> appears as if:
>
> There are 105 cases
> Length breakdown:
>
> Note: Previous runs of Timecheck3 and MaxLength=100 showed max
> length was 99 or 100 so I had upper bound.
>
> Length Number
> > 100 ...... 0
> 91 to 100 ... 17
> 81 to 90 ..... 8
> 71 to 80 ..... 9
> 61 to 70 ..... 12
> 51 to 60 ...... 9
> 41 to 50 ..... 10
> 31 to 40 ...... 7
> 21 to 30 ..... 15
> 16 to 20 ...... 6
> 10 to 15 ...... 2
> 8 or 9 ....... 2
> 6 or 7 ....... 2
> 5 ............ 1
> 4 ............ 2
> less than 4 ... 3
> total 105
>
> Also did a sum which checks how many short strings there are and can
> give a rough percentage of the two types of atoms
>
> Sum of elements Number
> > 70 ....... 2
> 61 to 70 ....... 0
> 51 to 60 ....... 5
> 41 to 50 ....... 21
> 31 to 40 ....... 16
> 21 to 30 ....... 18
> 11 to 20 ....... 22*
> 6 to 10 ....... 6
> 5 ....... 2
> 4 ....... 1
> 3 ....... 4
> 2 ....... 2
> 1 ....... 4
> 0 ....... 2
> total 105
>
> Note: On sum for 11 to 20 made typo so actually ran the same almost
> identical test twice once got 23 as answer, other time got 22 so
> picked 22 so total came out the same.
>
> I hope this can help people with their algorithms.
>
> Russ
>

Isn't it enough to start a new Matlab
session instead of rebooting?
That's faster and provides a clean start
for each entry also.


Matthew Simoneau wrote:
>
>
> Stijn Helsen wrote:
>> Didn't you reboot the PC everytime last contests ?
>
> Nope. We weren't rebooting the PC for every entry in earlier
> contests either.
>

Great Rumpuscat wrote:
>
>
> For the three strings that I believe I determined, I was able to
come
> up with better foldings than the front runners. But, somehow I
> wasn't able to take advantage of that information, which I find a
> little mystifying. Anyway, maybe this posting will be of use to
> someone.
Maybe that comes because the random number sequence is changed if you
replace a current solution by a fast one. (Refer to the improvement
by the "holy line" 'rand(63,1)'


Stijn

Stijn Helsen wrote:
>
>
> Great Rumpuscat wrote:
>>
>>
>> For the three strings that I believe I determined, I was able to
> come
>> up with better foldings than the front runners. But, somehow I
>> wasn't able to take advantage of that information, which I find
a
>> little mystifying. Anyway, maybe this posting will be of use to
>> someone.
> Maybe that comes because the random number sequence is changed if
you
> replace a current solution by a fast one. (Refer to the improvement
> by the "holy line" 'rand(63,1)'
>
> Stijn
>


I think that it can be useful to omit rand and randn, to make the run
and the score for the entries deterministic. It is easy to create
pseudo-random numbers if one like to use random search algorithms. It
doesn't count if the pseudo random numbers repeat, while they are
used only a few times. They can be computed from the given input of
the solver function or some random matrices included in the code. If
rand omitted even the least improvement in speed or cost can be
measured deterministic. It is possible to tune the pseudo-random
generator to get better results but if the generator is the same the
changes are always measurable


Sragner Laszlo

To make it deterministic you could just set the seed (or state) of
rand and randn. That is what I did when I wanted to get consistent
results when using random algorithms.


Russ


> I think that it can be useful to omit rand and randn, to make the
run
> and the score for the entries deterministic. It is easy to create
> pseudo-random numbers if one like to use random search algorithms.
It
> doesn't count if the pseudo random numbers repeat, while they are
> used only a few times. They can be computed from the given input of
> the solver function or some random matrices included in the code. If
> rand omitted even the least improvement in speed or cost can be
> measured deterministic. It is possible to tune the pseudo-random
> generator to get better results but if the generator is the same the
> changes are always measurable
>
> Sragner Laszlo
>

At the least the state of the rand and randn generators could be
reinitialized before each run. This would ensure that an identical
submission would score the same each time. It also plugs a potential
hole to send data between submissions -- the rand function has 32
integers of read/writ able state. (Also, setting the state of rand to
all zeros could also have bad consequences for future rand users.)
Maybe TMW contest organizers are already doing this.
Ken


Laszlo Sragner wrote:
>
>
> Stijn Helsen wrote:
>>
>>
>> Great Rumpuscat wrote:
>>>
>>>
>>> For the three strings that I believe I determined, I was
able to
>> come
>>> up with better foldings than the front runners. But,
somehow I
>>> wasn't able to take advantage of that information, which I
find
> a
>>> little mystifying. Anyway, maybe this posting will be of
use to
>>> someone.
>> Maybe that comes because the random number sequence is changed
if
> you
>> replace a current solution by a fast one. (Refer to the
improvement
>> by the "holy line" 'rand(63,1)'
>>
>> Stijn
>>
>
> I think that it can be useful to omit rand and randn, to make the
run
> and the score for the entries deterministic. It is easy to create
> pseudo-random numbers if one like to use random search algorithms.
It
> doesn't count if the pseudo random numbers repeat, while they are
> used only a few times. They can be computed from the given input of
> the solver function or some random matrices included in the code. If
> rand omitted even the least improvement in speed or cost can be
> measured deterministic. It is possible to tune the pseudo-random
> generator to get better results but if the generator is the same the
> changes are always measurable
>
> Sragner Laszlo
>

Yi Cao wrote:
>
>
> Is global variables are cleared after each call? The entry of
Peter's
> should be tuned cannot run on my computer because C is determined as
> a global variable. Therefore, second call will give a wrong size of
> C. However, it seems ok on the contest machine. So, the global
> variable C must be cleared after the first call.
>
Another possibility is that it can go well if the testcases are
sorted, so that the C-array can grow, and can never be to big.
That's what thought.
Stijn

globals are cleared between each testcase in the testsuite
Mike

Yi Cao wrote:

> Is global variables are cleared after each call? The entry of Peter's
> should be tuned cannot run on my computer because C is determined as
> a global variable. Therefore, second call will give a wrong size of
> C. However, it seems ok on the contest machine. So, the global
> variable C must be cleared after the first call.
>

It was possible to get the contest suit by using uninitialized
variables. Code a 10 long sequence of the protein into an integer,
then create a switch case for all the 1024 case of every possible 10
bit integer values and in every case an uninitialized variable named
by the code of the integer (of course fixed). If we get the first
sequence, at the next run we can identify it and go on with the next.
While the contest suit cannot be much more complex then the test
suit, the suit can be sequenced in hundreds of entries. I think that
permute the suit does not help while we identify every possible
sequences and then connect them to each other in the appropriate
order.


Sragner Laszlo

Well, it is the second contest in a row that I have a little black 2
in front of my name instead of that big red 1!


I agree that it was hard to keep up with the algorithms. The last
contest seemed much easier. But, I think we did a much better job
this time. I think the winning solutions last time leave room for a
lot of improvement. This time, I am not so sure. Maybe The
Mathworks will offer an opinion?
 
E. Brian Welch wrote:
>
>
> Hello Everyone,
>
> I hope I did not make anyone angry with my last minute cheat today.

Look again...we tossed that pesky hacker out of first

MR Keenan wrote:

> Well, it is the second contest in a row that I have a little black 2
> in front of my name instead of that big red 1!
>
>
> I agree that it was hard to keep up with the algorithms. The last
> contest seemed much easier. But, I think we did a much better job
> this time. I think the winning solutions last time leave room for a
> lot of improvement. This time, I am not so sure. Maybe The
> Mathworks will offer an opinion?
>
> E. Brian Welch wrote:
>
>>
>>Hello Everyone,
>>
>>I hope I did not make anyone angry with my last minute cheat today.
>>

The contest was fun, the setting of the
time/results constants are also adequate.


I like the fact that the winner was the
one who at the last improved the search result. Certainly the winning
entry can
be improved in speed, at least we can remove several "special cases"
that were not longer needed. Also some variables as the locations of
ones, length of ones, length(a), can be computed once and not in
every added function.


For me, one of the greatest contributions was 'searchnfold', it was
fast and well preogrammed.


It's been fun. Congratulations to all of you which wrote some part of
code in the winning entry and also to Mr.Keneen.


MR Keenan wrote:
>
>
> Well, it is the second contest in a row that I have a little black 2
> in front of my name instead of that big red 1!
>
> I agree that it was hard to keep up with the algorithms. The last
> contest seemed much easier. But, I think we did a much better job
> this time. I think the winning solutions last time leave room for a
> lot of improvement. This time, I am not so sure. Maybe The
> Mathworks will offer an opinion?
>
> E. Brian Welch wrote:
>>
>>
>> Hello Everyone,
>>
>> I hope I did not make anyone angry with my last minute cheat
today.
>

With the release of the scoring testsuite, I confirmed that the input
sequences I reported in Msg#68 were correct (length 10,34,36 are
entries 64,63,92 respectively) . It was interesting to see this
information used for many entries up until the very end, and then it
was dropped. I'm glad the winning entries didn't use it, actually.
It's also interesting how unhelpful this information actually proved
to be.


While the CPUtime-as-communication-channel method could be used to
decode the whole testsuite, it is very tedious. However, someone
else submitted a single entry near the end of the contest in which
they got a whole input string in one submission. It is very clever.
(There is a small error in the decoder which can be fixed). E. Brian
Welch also tried to use that method, but made a error which he didn't
have time to fix, apparently because he had even more clever ways to
win. This method could easily be generalized.


Ken


Michael Thomas wrote:
>
>
> The zip file on the File Exchange has been updated with the
testsuite
> used to score the contest.

Great Rumpuscat wrote:
> However, someone
> else submitted a single entry near the end of the contest in which
> they got a whole input string in one submission. It is very clever.
> (There is a small error in the decoder which can be fixed).


Which entry was this? I'd like to check it out.


Thanks.

As I concerned, at least there are two places we can improve. One is
the minenergy function. The idea using variance to quickly pick up
best result is great. However, it still puzzles me that if it is
always correct why don't we remove sum of abs completely to improve
speed; if not, then we may not get the best. Is there any condition
exists for best var equivalent to best sum of abs? The second is the
efficiency of searchnfold code. The code is effective to find good
folding however code is not so efficient. I am sure we can improve
it. The consequence is that we can then increase the search depth
within the same CPU time.


MR Keenan wrote:
> This time, I am not so sure. Maybe The
> Mathworks will offer an opinion?
>

Matthew Simoneau wrote:
>
>
> Great Rumpuscat wrote:
>> However, someone
>> else submitted a single entry near the end of the contest in
which
>> they got a whole input string in one submission. It is very
clever.
>> (There is a small error in the decoder which can be fixed).
>
> Which entry was this? I'd like to check it out.


I submitted the "error test" entry which contains a decoder with an
error. See the 'determining the series in the contest suite' thread
for a correction of the decoder.


Bert

Bert Jagers wrote:
> I submitted the "error test" entry which contains a decoder with an
> error. See the 'determining the series in the contest suite' thread
> for a correction of the decoder.


Very clever! This would have worked. This method would have made it
easy to determine all of the tests in the suite.


Now the question is, can you think of a way for us to plug it? The
only sure way I can think of is to not show error messages, but this
would make it harder for contestants to debug their entries. Can you
think of any other way?


References:


<http://www.mathworks.com/contest/protein.cgi/view_submission.html?
id=10143>


<Bert Jagers "determining the series
in the contest suite" 11/14/02 12:31pm>

An easy way to plug it is to let an entry go through a random test
before sending it to a formal test suit. Not only can the random test
quickly spot an error for genuine debug purpose but also protect real
test suit to be identified.


Matthew Simoneau wrote:
>
> Now the question is, can you think of a way for us to plug it? The
> only sure way I can think of is to not show error messages, but this
> would make it harder for contestants to debug their entries. Can
you
> think of any other way?
>
> References:
>
>
<http://www.mathworks.com/contest/protein.cgi/view_submission.html?
> id=10143>
>
> <<a href="/WebX?50@@.eeb51da">Bert Jagers "determining the
series
> in the contest suite" 11/14/02 12:31pm</a>>
>

Another way to plug it....


What about changing the order of the test suit for every entry, this
shouldn't impact the overall performance.

Lucio wrote:
>
>
> Another way to plug it....
>
> What about changing the order of the test suit for every entry, this
> shouldn't impact the overall performance.
>


This solution wouldn't stop the method I described. Okay, one would
not be able to know the order in which the problems would be offered
to the solver, but one would still be able use the method to
determine all the cases in the following manner:


if isequal(a,predefinedcase1)
elseif isequal(a,predefinedcase2)
elseif isequal(a,predefinedcase3)
...
else
  generate error that indicates the
  unhandled case
end


This method would, however, make the solvers less sensitive to
particular series of rand results. Which makes the comparison between
different implementations fairer. On the otherhand, a
non-deterministic solver (i.e. one calling rand) would give a
different result each time it was submitted. The method for computing
the score should be adapted accordingly. Several options were
discussed earlier.


Bert

Yi Cao wrote:
>
>
> An easy way to plug it is to let an entry go through a random test
> before sending it to a formal test suit. Not only can the random
test
> quickly spot an error for genuine debug purpose but also protect
real
> test suit to be identified.


If you are referring to a really random testcase, this would indeed
work. But then one should also clear globals between successive calls
of the solver, otherwise one would be able to skip the first n calls
by using a simple solver like the one initially submitted by The
Mathworks.


However, if you are referring to a random pick from the training or
test set, this will not help since the number of cases to detect
remains the same. Or, if you are referring to a random pick from any
other set of relatively small size, this will only take slightly
longer because some cases have to be determined.


Bert

Matthew Simoneau wrote:
>
> Now the question is, can you think of a
> way for us to plug it? The
> only sure way I can think of is to not
> show error messages, but this
> would make it harder for contestants to
> debug their entries. Can you
> think of any other way?
>


The plug suggested by Yi Cao by using a really random case as a
filter for the submitted entries, might indeed be the best solution.


One should, however, note that only the Mastermind and Protein
contests have been sensitive to this cheat. The other recent contests
use much larger input variables (a set of real values, a large matrix
of 0/1's) which are more difficult (maybe impossible) to encode in
the 31 characters of a valid fieldname (has this been extended to 63
together with the variable name length in Matlab 6.5?).


Otherwise, one would have to resort to a filter operation on the
error messages, such that the fieldname is not shown. This would,
however, be less informative in case of real errors.


Bert

Obviously, global variables have been cleared after each call to the
solver since the global hole was spotted. I noticed it because I
could not run some top entry code in my own computer. I found this is
because the code has an error using a global variable in the way,
C(:,i)=c without redefining correct size for C. Since these code can
pass master test, the master must clear global variables between each
call to solver. Someone suggested that the master test may have
sorted in size so that C can gradually expand without error. However,
after we see the master testsuit, this is not the case.


Bert Jagers wrote:
> work. But then one should also clear globals between successive
calls
> of the solver, otherwise one would be

Firstly, thanks for your comments on the searchnfold algorithm. Sorry
it wasn't commented - like most other contestants, I was too busy
coding to write about it :) Having said that though, it's not a good
excuse, and I do think we should in general, endeavour to make our
code more readable before submitting it. For those of you that had
difficulty catching up, don't be disheartened, we're all in the same
boat. In fact, it usually takes me a good part of the first day just
to figure out what all the other algorithms are doing! I did actually
write a second version of searchnfold (seen in the ffsearch series)
which was much more efficient, and did very well on my homegrown
testsuite beating the leading entry by about 6 score-points!
Unfortunately, it wasn't tuned (or streamlined) for the actual
testsuite and so performed miserably in the contest :( I did however
manage to use the code to find a better solution for the length-36
string found by Ken Crounse, so it wasn't completely useless...


On that same note, I created a second 'homegrown' testsuite, based on
the statistics discovered by Russell Newcomb and others, which is
what I used for most of my offline testing. It obviously wasn't up to
scratch though, given the discrepancy in my results compared to those
yielded by the contest server!


A couple of thoughts. What do people think about the idea of having
another (or possibly several other) category for 'best performance
(results) within the allotted time'? This would place more emphasis
on the performance of algorithms, rather than speed. Naturally speed
would still play a part, but I imagine that the resulting code would
be significantly different from the 'best speed + performance
(score)' stream. Also, would it be possible to implement a
'beginners' stream to make it easier for others to compete, or is
this just over-complicating the contest?...


I really like some of the ideas Stefan Stoll suggested and I think it
would be great to see more scope for collaboration and offline (or
non-contest) testing. Avoidance of spaghetti-code is another
important issue (made worse I think, unfortunately, by the new
acceleration capability and the increased use of loops and other
nasties :)), but a difficult one to tackle. Are p-coded files
independent of the lengths of variable names?


Another suggestion is a bookmark facility for selected entries. That
way, a list of each (web) user's favorites could be easily accessed.


Well done to Mike Keenan, Yi Cao and all the competitors.


See you all in the next round!


Roger

Will there be an analysis on the winning entry?

Hi Martin,


We'll be publishing a final contest analysis later this week. You'll
also receive e-mail notification if you're on our contest
announcements mailing list. See page below for detail on how to sign
up:


 <http://www.mathworks.com/contest/protein.cgi/home.html>


Meanwhile have you also checked out our contest winners page?


 <http://www.mathworks.com/contest/protein.cgi/winners.html>


-Min


Martin Law wrote:
>
>
> Will there be an analysis on the winning entry?
>