buggnubg - Bugs: bug #58608, Distribution of rolls contradicts...

 
 

bug #58608: Distribution of rolls contradicts hint evaluation?

Submitted by:  Tom Martin <tom_martin>
Submitted on:  Wed 17 Jun 2020 04:24:30 PM UTC  
 
Originator Name:  Tom Martin Item Group:  Other bug
Category:  Evaluations Release:  1.04-Release
Privacy:  Public Operating System:  Windows 8
Severity:  3 - Normal Open/Closed:  Open
Assigned to:  None Priority:  5 - Normal
Status:  Confirmed

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Wed 29 Jul 2020 10:30:28 PM UTC, comment #8: 

Thanks, Christian.  I should have realized that cube handling must have somehow been responsible, especially with the big difference in equity despite similar raw numbers.  As is probably common in such cases, a rollout tells a very different story.

Curiously, if one knows (or deduces) the cause, the current output is probably more valuable.  It shows the relative rankings of the rolls without any cube "distortion" and lets you know that there's significant cube vig involved.  That tells you that a rollout is needed to confirm the hint and (perhaps more importantly) that a small change in the position will often result in very different equities.  It's important that players be on the alert for situations where evaluations are being greatly affected by the cube.

Tom Martin <tom_martin>
Sat 25 Jul 2020 04:53:59 PM UTC, comment #7: 

I've played around with this a bit and my conclusions are

1. neither evaluate nor distribution of rolls does exactly what one would expect.

2. the main problem is with distribution of rolls, which doesn't update the cube. Instead the cube is frozen and the evaluation is based on cubeful moves and cubeful evaluations on the final ply

3. If you want to play around with this further you can use truncated rollouts with expert (0-ply) evaluations and compare to evaluations at the same truncation (ply) level. Maybe turn off pruning in the evaluations

Christian Anthon <c_anthon>
Project Administrator
Fri 26 Jun 2020 12:34:56 AM UTC, comment #6: 

You knew I'd post again, right?  But naturally with only good news - an analysis of METs shows that doubles at this score can't be taken quite as deeply as for money.  This could easily explain why the aggressive play is wrong ATS (although the massive differences reported by 3-ply (and 4-ply) are still surprising).

So the only remaining question is whether I should expect the distribution of rolls to support the Hint's evaluation.  Simply put:  if the equity is a certain value (.978 here), why might none of the possible rolls be worse than that?

Tom Martin <tom_martin>
Thu 25 Jun 2020 10:19:07 PM UTC, comment #5: 

Sorry - ignore the final post.  Winning gammons only changes relevance when WE are 3-away.  So I'm still struggling to see why winning gammons is of less value at 6-away, 3-away.  And why the distribution of rolls doesn't seem to support the evaluations.

Tom Martin <tom_martin>
Thu 25 Jun 2020 08:36:43 PM UTC, comment #4: 

One last oddity and then I'll pipe down.  I figured hitting was not as attractive as at unlimited play since my opponent is at 3-away and thus we can never score 4 points.  So I changed it to 4-away and the bot said hitting was even worse!

This seems like a guarantee that something is wrong with its evaluations.

Tom Martin <tom_martin>
Thu 18 Jun 2020 01:27:06 AM UTC, comment #3: 
      • please read ***

My apologies - apparently I was just assuming that the 2-ply and 3-ply equities must be very different.  When I verified that they weren't, it explains what I observed in my last post but reopens the case.  If you change the hint to 3-ply, you'll see the equity is .978 for the top play.  If you then make the play, let the bot move, click on its move to reset the position, click Analyze, then Distribution of Rolls, you'll see that even the worst possible rolls (for both 2-ply and 3-ply) are better than -.978.  Only 1-ply shows the bad equities.

Tom Martin <tom_martin>
Wed 17 Jun 2020 05:24:42 PM UTC, comment #2: 

Another odd thing (feel free to ignore this) is that after playing the suggested move, I went into Evaluate and it showed virtually no difference between the three ply levels.

The key is probably something about match play.  If you look at the raw numbers in the original position, there's nothing to suggest much of a difference between the top two plays.  Yet the equity is much higher  for the safe play.  In money games, the evaluation is reversed.  Tim Chow suggested that this was due to badly missing the market in a match situation, but that seems odd to me.

Tom Martin <tom_martin>
Wed 17 Jun 2020 04:51:16 PM UTC, comment #1: 

Never mind - when comparing a hint's equities to the distribution of rolls you must always use a ply of ONE LESS than the hint.  In this case, the 2-ply and 3-ply analysis greatly differs.

Curiously, even when you adjust the ply count, the average equity rarely exactly matches the hint's equity.  But I suppose that must be because the hint is not designed to show the "average equity".

Tom Martin <tom_martin>
Wed 17 Jun 2020 04:24:30 PM UTC, original submission:  

I was skeptical of the 2-ply hint's evaluation so I played the suggested move and then looked at the distribution of rolls.  Sure enough, they didn't support what the hint showed.  I suppose I must be missing something but I'm clueless as to what it might be.

XGID=-B-DAaB-D---ba-bAbbe---A--:0:0:1:43:1:4:0:7:10

In the two screenprints, note the .979 evaluation for 23/16.

Tom Martin <tom_martin>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by c_anthon (Posted a comment)
  • -email is unavailable- added by tom_martin (Submitted the item)
  • -email is unavailable- added by tom_martin
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

     

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-07-25 c_anthon StatusNone => Confirmed
    2020-06-17 tom_martin Attached File- => Added Screenshot@(811).png, #49299
        Attached File- => Added Screenshot@(810).png, #49300
        Carbon-Copy- => Added shphilacintiwc@gmail.com

    Back to the top


    Powered by Savane 3.5