SudoCue Users

Ruud

In the forum on www.sudoku.org.uk you have posted a message rating some of their killers and including ratings for some recent Assassins.

It would be interesting if you could post a similar rating here for all the Assassins, except any currently active, and possibly some of the posted variants and other puzzles posted on this forum.

We could then see how these ratings compare with our own personal experiences of which were the hardest puzzles. It would also show whether Assassins have generally been getting harder, which I think they probably have.

Your rating counted the number of advanced steps and also the level of the most advanced step required. If you produce a list here, it would be interesting to know which advanced steps count for each of the levels.

I would also suggest that, if the list is posted, it should be as a one message topic with a sticky. Therefore if Ruud contacts me before posting a list, I'll delete this message. Any discussion of the list would probably be best done on a separate topic.

Andrew,

I see Ruud hasn't answered your post (yet), but I'm already wondering whether it's feasible to rate all assassins, as you suggest, because a program can surely only accurately rate puzzles it can solve, and several of the newer assassins are AFAIK not solvable by SumoCue.

Maybe Ruud can give some more input on this.

I will assemble and post a list of ratings for the Assassins so far, but don't be surprised when many of them are "off the scale". When SumoCue cannot complete an Assassin, I often copy the semi-finished puzzle between several programs to find additional steps, which are not reflected in the rating.

Keep these walkthroughs coming! They are my primary source for selecting additional techniques to implement in SumoCue.

cheers,
Ruud

Andrew wrote: if you could post a similar rating here for all the Assassins...We could then see how these ratings compare with our own personal experiences of which were the hardest puzzles

Getting closer to a numerical, software generated version of this Andrew...but want to get some feedback on a smaller selection first - all from the A50+ Ratings Sticky.

Richard released (on Sunday) SudokuSolver V2.1.2[edit to correct V number], available in the software forum, which includes what I call an SSscore. This software generated score tries to predict what human rating we, on the Assassin forum, end up giving to a killer after we solve it. Here is a summary of the SSscores for all the puzzles that are currently on the Ratings Sticky.

I have also included a suggested new rating for 14 of the puzzles that are on the Sticky. This is to bring those 14 puzzles closer into line with the SSscore AND still reflect the general difficulty that we human solvers commented on. If there was general consensus about the rating for a puzzle then it should not be one of the 14. But if there were a range of human ratings given, or none, then I propose moving these 14 to a rating category that better fits the SSscore.

I also hope to re-organize the Sticky to give a more accurate sense of hardest to easiest in each rating category. The 1.75 rating category, for example, appears to include a huge range of difficulty level (based on the SSscores).

Another reason for the change of rating category is that it raises the correlation between the predicted SSscore and the ratings from 85% to 89% for (.125) rating accuracy. This correlation is from a group of 130 killers including (most) Assassin, Assassin forum and a few other representative puzzles from DJApe, Michael Mepham's Diabolical killer's and a Times Deadly. All of these 130 get an SSscore below 2.0.

HOWEVER - I have had to guesstimate the rating for each of these puzzles not on the Sticky in order for Richard to develop/find a scoring routine which could predict these ratings. If it appears that the SSscore is reasonably accurate, then I will post the complete list, and hope to use it to increase the Rating sticky to include Assassins 1-24 and 25-49.

Just to restate - I would really like some specific feedback, especially objections/concerns about moving these 14 puzzles (3rd 'column' & in bold - thanks Mike). And of course, specific feedback about if you feel we are on the right track would be really appreciated.

Can someone put this list into nice wide columns for me please, or send me a PM with how to do it? Thanks.

Was Cheerful
Ed

.......Sticky Rating-- SSscore-- New Rating?
A01- 0.75 0.50....................0.50
A01 (V2)- 1.25 1.25

A48 Hevvie- 2.50 2.63

A50- 1.75 2.13
A50 (V0.2)- 0.75 0.63
A50 (V2)- 3.00 3.00

A51- 1.25 1.38

A52- 1.25 1.25
A52 (V2)- 2.00 2.00

A53- 1.00 1.00
A53 (V01)- 2.00 2.38
A53 (V2)- 1.25 1.00..............1.00
A53 (V25)- 1.25 1.25
A53 (V3)- 1.75 1.75

A54- 1.25 1.13
A54 (V2)- 1.75 1.50..............1.50

A55- 1.25 1.25
A55 (V2)- 2.00 2.13

A56- 1.25 1.00...................1.00
A56 (V2)- 1.75 2.13

A57- 1.00 0.88
A57 (v15)- 1.00 1.25..............1.25
A57 V2- 1.75 1.63

A58- 1.00 1.13
A58 (V15)- 1.50 1.13.............1.25

A59- 1.25 0.88
A59 (V15)- 1.50 1.63

A60- 1.50 1.50
A60 RP- 3.00 3.13
A60 RP-Lite- 1.75 2.50

Special X
Killer 4 (V2)- 4.0 no score

A61- 1.25 0.88................1.00
A61X- 1.50 1.50
A61X V3- 4.0 no score

A62- 0.75 1.0
A62 (V2)- 2.00 2.25

A63- 0.75 1.0
A63 (V15)- 1.00 1.38
A63 V2- 1.50 1.25..............1.25

A64- 1.25 1.13
A64 (V2)- 1.75 2.25

A65- 1.25 1.25
A65 (V2)- 1.75 1.75
A65 (V3)- 1.75 1.75

A66- 1.00 1.25
A66 (V1.5)- 1.50 1.88.................1.75

A67- 0.75 1.00

Vortex X- 1.75 1.25................1.50
Vortex X Lite- 1.25 0.88

Trans X- 1.75 2.13
Trans X Lite- 1.25 1.37

A68- 1.00 1.25....................1.25
A68 (V1.5)- 1.25 1.50..............1.50
A68 (V2)- 1.75 1.75
A68 (V3)- 3.00 2.88

A69- 1.25 1.38
A69 (V1.5)- 1.50 1.63

A70- 0.75 1.13
A70 (V2)- 1.75 1.38
A70 (V3)- 1.25 1.38

A71- 1.75 1.13.......................1.50
A71 (V1.5)- 1.25 1.75..................1.50
A71 Full
Border- 1.50 1.25

CS- 1.25 1.13

A72 - 1.00 1.38
A72 Old School- 1.75 2.63

A73 - 1.00 1.00

Maverick (1)- 1.75 1.25

A74- 1.00 1.25
A74
Brick Wall- 2.00 4.50
A74 (V2)- 1.25 1.25

A75- 1.25 1.50

A76- 1.00 1.13
A76X- 1.25 1.38

Diagonal
Surprise- 1.25 1.0

DJape
Easy (6/11)- 0.50 0.50

DJape
Brain (8/10/06)- 0.75 0.75

DJape
IQ (17/11)- 0.88 1.0

DJape
Insane (6/11/06)- 1.25 1.38

MM Online
Diab (19-11-07)- 0.63 0.63

Times
Deadly (673)- 0.63 0.63

Hi Ed,

Thanks for the status update. I get the feeling that this thread is going to get quite a few new posts in the near future...

sudokuEd wrote:I would really like some specific feedback

Hmm, don't know what Andrew's going to say about SudokuSolver giving the A74 the same rating (read: SScore) as the M1! It proved impossible to convince Andrew about downgrading the M1 from 1.75 to 1.5, let alone making a 1.25 out of it, as is the case here!

That said, I notice that this puzzle is apparently not one of the 14. BTW, these 14 are the ones with three columns of numbers, right? Maybe it would be a good idea to highlight these (e.g., color or bold type) so that they are visible at a glance?

On the postive side, many ratings seem to be quite convincing, like the 1.38 for the A76X, which is mid-way between my estimation (1.5) and Para's and Afmob's verdicts (1.25). The Diagonal Surprise at 1.0 is probably also realistic. I think Andrew and I only rated it higher because (in contrast to machines) humans tend to have more of a problem with diagonal cages (although they are child's play in comparison to Para's toroidal jigsaws... ).

Keep up the good work!

Mike wrote:Hmm, don't know what Andrew's going to say about SudokuSolver giving the A74 the same rating (read: SScore) as the M1! It proved impossible to convince Andrew about downgrading the M1 from 1.75 to 1.5, let alone making a 1.25 out of it, as is the case here!

I had to check back to on A74 and I see that I commented "This one flowed easily for quite a time, then got a bit stubborn so I'll rate it at 1.0". Clearly I had considered rating it lower. That would seem to suggest that the SS score of 1.25 for it is dubious.

I seem to remember something about SS finding a shortcut for M1 although I can't remember the details. That would explain it's absurdly low rating. Ratings shouldn't be based on shortcuts if there are normal solutions available which don't need hypotheticals or T&E. Mike has, in the past, stated that the rating should be based on the best human solution, which I take to be the best posted one. Clearly best doesn't includes solutions using shortcuts unless they are essential which, in the case of M1, they weren't.

Mike wrote:The Diagonal Surprise at 1.0 is probably also realistic. I think Andrew and I only rated it higher because (in contrast to machines) humans tend to have more of a problem with diagonal cages.

When I posted my walkthrough

I wrote:I agree with Mike's rating of 1.25. Diagonal cages are always harder for human solvers so, for that reason, I won't rate it any lower.

For that reason I feel it should remain at 1.25.

Ed wrote:A71- 1.75 1.13.......................1.50
A71 (V1.5)- 1.25 1.75..................1.50
A71 Full
Border- 1.50 1.25

Are you sure you've given the SS scores for these three in the right order?

It was generally agreed that A71 was the hardest ever original Assassin and that both variants were easier than the original. I see that I commented for V1.5 "I'll rate it at least a 1.25. Maybe it should be rated higher because it was a fairly narrow solving path and a lot of work after the breakthrough." Maybe therefore the 1.25 rating for V1.5 is low but it wasn't a 1.75.

The only other ones I will comment on are that the SS scores for Vortex X and Vortex X Lite look far too low. Maybe more cases of SS using shortcuts? I just don't believe those two scores.

I've commented before about each rating number representing a range of difficulty. However I'm not sure if I've commented that when humans rate puzzles they probably also take account of whether they found steps quickly or took a long time to find them; this may be done consciously or without realised that it has been done. It's not just the difficulty of steps but also the ease or difficulty of finding them.

Andrew wrote:I seem to remember something about SS finding a shortcut for M1 although I can't remember the details. That would explain it's absurdly low rating. Ratings shouldn't be based on shortcuts if there are normal solutions available which don't need hypotheticals or T&E. Mike has, in the past, stated that the rating should be based on the best human solution, which I take to be the best posted one. Clearly best doesn't includes solutions using shortcuts unless they are essential which, in the case of M1, they weren't.

The only other ones I will comment on are that the SS scores for Vortex X and Vortex X Lite look far too low. Maybe more cases of SS using shortcuts? I just don't believe those two scores.

I've commented before about each rating number representing a range of difficulty. However I'm not sure if I've commented that when humans rate puzzles they probably also take account of whether they found steps quickly or took a long time to find them; this may be done consciously or without realised that it has been done. It's not just the difficulty of steps but also the ease or difficulty of finding them.

It is always going to be problematical using a program to produce reliable preditions of how a human is going to rate a puzzle.

I agree with you Andrew that a human solver adds other factors into the perceived complexity of a puzzle - I can sometimes stare at a puzzle for hours before spotting an "obvious" move and so may consider it more difficult than it deserves. Equally, if I spot a key move early on - regardless of how "complex" it may be then the puzzle "feels" easier.

In the rewrite of some of the routines in SudokuSolver to try to get a better correlation to human percieved complexity, I've simplified a number of routines to ty to make them more "human" and along the way introduced a couple of new tricks that the forum has pointed out to me. The former now means that there should be more understandable moves earlier on and only getting to really convoluted combinations when the simple moves fail to turn up eliminations.

The other result is that there are a lot fewer T&E steps turning up in SS solver logs now - of the 135+ puzzles Ed and I have worked on, the only ones now requiring T&E are A39V2, A41V2, A55V2, A72V2, A74BW and A60RP.

But despite all that, there are still some real surprise in there. As you picked out - A71 is a great example. However many times I tried to persuade the program that the original A71 was harder than the variants, it stubbornly resfused to agree with me and consistently solves A71 much quicker, with fewer steps than either of the variants. I'm still sifting through some of the solver logs to see if there are any hidden complex moves that are not getting correctly factored - but there's nothing jumping out at me.

When all is said and done, the solver just mechanically applies a set of rules over the problem and Ed and I have calculated the best weighting to apply to each rule to get a close correlation to the perceived scores. Whilst we can use this to pose some questions, and maybe highlight some puzzles with "back-doors" that lead to quicker solutions - it's never going to replace the human factor. And much as I'm quite proud of the programming in SudokuSolver, I don't think I'm ever going to be able to give it enough intelligence to really rate puzzles like a human would.

Rgds
Richard

Hi guys,

I've now gone through Ed's posting in more detail, and tried to identify the ratings (sorry, SScores...) in most need of attention. There were 20 I could see where the SScores deviated substantially enough from the perceived ratings in order to potentially receive special attention. Here are my initial comments on these.

I wrote:A50- 1.75 2.13
Admittedly too high, but chain needed by Para and Cathy required UR move (BTW - does the SScore consider UR-based moves?). So the high SScore is not too worrying here.

A56 (V2)- 1.75 2.13
SScore is within the bounds of reasonability here.

A58 (V15)- 1.50 1.13
Agree with downgrading to 1.25.

A59- 1.25 0.88
SScore still a problem. Needs further investigation.

A60 RP-Lite- 1.75 2.50
SScore is OK. Puzzle needed some very complex moves.

A61- 1.25 0.88
SScore still a problem. Needs further investigation.

A63 (V15)- 1.00 1.38
This was one of mine. I was surprised by the low rating of 1.0 at the time, so the difference here is possibly OK.
Suggest upgrading to 1.25.

Andrew wrote: "For [the] V1.5 it would probably be rated as... 1.25 for the way that I solved it."

A64 (V2)- 1.75 2.25
Team effort used forcing chain from Glyn (step 54), so SScore OK.

A66 (V1.5)- 1.50 1.88
SScore reasonable. Puzzle required "fancy" combination work from Para.

Vortex X- 1.75 1.25
SScore still a problem. Needs further investigation.

Vortex X Lite- 1.25 0.88
SScore still a problem. Needs further investigation.

Trans X- 1.75 2.13
This one contained a "trick", in view of which the rating is possibly not too outlandlish. May need attention later, but not one of the "urgent" cases for now.

A70- 0.75 1.13
SScore still a problem. Needs further investigation.

A70 (V2)- 1.75 1.38
SScore still a problem. Needs further investigation.

A71- 1.75 1.13
SScore still a problem. Needs further investigation.

A71 (V1.5)- 1.25 1.75
1.25 rating is possibly too low. Agree with upgrade to 1.5.

Andrew wrote: "I'll rate it at least a 1.25. Maybe it should be rated higher because it was a fairly narrow solving path and a lot of work after the breakthrough...I had to use one hypothetical..."

A72 - 1.00 1.38
SScore still a problem. Needs further investigation.

A72 Old School- 1.75 2.63
I used a shortcut to solve it. 2.63 is probably overrated, but aim of SScore V1 should probably be to get the ratings under 2.0 reasonably accurate.

Maverick (1)- 1.75 1.25
SScore still a problem. Needs further investigation.

A74 Brick Wall- 2.00 4.50
SS probably missing several tricks? Probably not so urgent, because (as mentioned above) initial aim should be to get the SScores for the non-extreme puzzles right.

Of course, this is all only my personal view. But it shows that IMO only 9 puzzles in the list require "urgent" (detailed) attention, which is a very encouraging result! No rating scheme can ever be perfect, but the results obtained up to now are already better than I originally expected them to be. Thanks, Ed and Richard!

Whether all the deviations highlighted in red above need to be fixed prior to release is open to question. But at least it would be nice to have an explanation for the difference in each case.

BTW, last but not least, I've just noticed that no less than 11 of the above 20 puzzles are from me or Para. Seems like we're especially good at keeping SudokuSolver on its toes...

Good to see your post Richard. It seems quite a long time since we've heard from you.

It's interesting to read your views on the ratings. You and Ed are doing a great job. Without your continued improvements to SS there wouldn't be any ratings apart from the human estimates. Many thanks to both of you! Thanks also to Mike for formulating the original rating definitions.

Andrew wrote:Good to see your post Richard. It seems quite a long time since we've heard from you.

It's interesting to read your views on the ratings. You and Ed are doing a great job. Without your continued improvements to SS there wouldn't be any ratings apart from the human estimates. Many thanks to both of you! Thanks also to Mike for formulating the original rating definitions.

I've been a bit busy keeping on top of modifications to the solver - one of these days, Ed and I will get back to solving, if I can remember how .

Ed has done a tremendous job pushing on with the ratings work and prompting a lot of the changes to the program.

We are a lot closer on a lot of the ratings than I thought we'd ever get and, who knows, we may yet get a breakthrough on the problematic puzzles. More work analysing the solver logs compared to the forum walkthroughs and looking for any additional routines or simplifications and we may yet get there.

Rgds
Richard

I've went through the solver log of A71 and i think i can see what the problem is.
Firstly there's a really narrow solving path, even for Sudoku Solver. Secondly there are amazingly many contradictions between C7 and N6 and Sudoku Solver does this unbelievably effective. It took me loads of steps and time to accurately analyze these contradictions but Sudoku Solver does them really effectively and with some of its "basic techniques". After this it goes about in a really effective way through the puzzle. It just seems that this puzzle exactly fits Sudoku Solvers way of solving a puzzle. It will probably always stay a fluke.


Para

One thing that I don't think has yet been discussed is how we interpret each range.

For example Para and I gave different ratings for A77. Here are a few quotes, with Para's permission to quote from our PMs.

Andrew wrote:You will see that I've suggested that A77 is an easier 1.25 rather than 1.0, even the way you did it which isn't much quicker than my way.

Para wrote:I guess it depends on how you see the rating range. For me it is a difficult 1.0, something around a 1.20 and for me this falls in the 1.0 rating range.

Andrew wrote:I've always assumed that any puzzle that comes between two rating values is rated at the one that is closest so, for me, 1.2 is an easier 1.25. You clearly take a different view and I know that Ed does too. I've no idea which view other forum members take.

It could explain a lot of the discussion about the rating of some puzzles.

Actually I'm not sure about that last sentence now. On further thought people can have a different personal view of a puzzle's rating and still assign it the same rating value. For example a hard 1.25 might be 1.35 to me, because I consider 1.25 to cover the range 1.13 to 1.37 (make that 1.15 to 1.35 in practice) while someone who considers the rating value 1.25 to cover 1.25 to 1.49 might rate the same puzzle as 1.45. Both of us would rate this puzzle as a hard 1.25.

I must admit I was surprised when Ed first told me he took each rating range as extending from the rating value right up to the next rating value, for example that a rating value of 1.25 represented 1.25 to 1.49. It had started from a discussion about what an estimate of 1.0 to 1.25 meant. Ed said it must mean 1.0 because 1.0 to 1.24 was in that range while I said it meant that it could equally well mean either 1.0 or 1.25. I'd assumed, from when the ratings appeared, that each rating represented a range around the rating value so that one formed an opinion on the rating of a puzzle and then rounded to the nearest rating value. One thing is clear. When quoting a rating for a puzzle it's best just to give one value rather than a range; at least that removes one possible source of misunderstanding.

When I asked Para's permission to quote from our PMs he replied

Para wrote:No problem you might want to quote my A78V2 quote as well . I explained my reasoning for 1.5 there as well.
I mean otherwise A60RP-lite might get a 2.0 as well as i see that as a firm 1.90.

Well no need to quote from Para's A78V2 walkthrough message since it's been done there.

The important thing is that Para is confirming that A60RP-lite is at the top end of the 1.75 rating range, rather than just being a typical 1.75. That's also confirmed in Ed's excellent rating sticky where it is listed as the first in the 1.75 rating group, indicating that it's the hardest in that group.

It's interesting to see in the original definitions for the rating values

Mike wrote:...The A60RP-Lite could maybe deserve such a rating.

An interesting "maybe"!

I better get a flurry on . Congratulations Mike (IA). Thoroughly deserved.

Andrew wrote: for me, 1.2 is an easier 1.25. You (Para) clearly take a different view and I know that Ed does too. It could explain a lot of the discussion about the rating of some puzzles

I also have doubts about your last statement Andrew. Our ratings are an approximation. On most puzzles, there is a reasonable consensus about the difficulty level. I think this is in large part due to Mike's original suggestion of just 6 different levels of difficulty for the 'normal' assassin puzzler (0.5 - 1.75). The validity of our 'gut feel' rating is shown by the fact that Richard and I have been able to get higher and higher correlation between the SSscore and forum ratings - at least up to low 1.75. After that, each 0.01 of Para's ratings is about a 10 added to the SSscore. More on that another time.

Lead-in to Assassin 78, Ruud wrote: Richard's SudokuSolver is still running while I'm writing this

Ruud must type fast, or has SudokuSolver V1. Part of Richard's upgrade to V2, which he has been too modest to mention, is that it solves (scores) up to 450 times quicker (A52: from 78 seconds to 0.17 sec)! An incredible achievement by Richard!

BTW - for those that use SudokuSolver, under "options" menu, down the bottom of the window is a "Step Delay" option. It has 6 different speeds, with the default setting on medium. I've put into my Christmas wish-list for Richard to change the default to "none". Occasionally I like the medium pace - something hypnotic, soothing and comforting about seeing SS show off those pretty colours while it powers along.

About A78V2 rating Para wrote: i actually want to rate it something like a 1.65....I know i am way of what Sudoku Solver thinks

I think I'm right that SudokuSolver V2.1.2, the current released version, scores it as 1.63. You have to make sure you go to the options and click on the "Scoring" routine before "scoring". Incidentally, the current scoring version 2 Richard and I are working on scores it higher than this at 1.87. This again shows why we need to keep some distinction between the human 'ratings' and the mechanical version. If the humans are agreed that this puzzle doesn't have that extra 'trick' essential to a 1.75 rating puzzle, then the SSscore is a good indicator of.....something else yet-to-be-discussed-at-some-length.

I haven't commented on the excellent feedback from Mike, Andrew and Para about the SSscores V1. Richard is doing another lot of changes to SS - but even before these we are getting higher correlations. So, will be back another time with updates.

Cheers
Ed

Just a comment on one point

sudokuEd wrote:If the humans are agreed that this puzzle doesn't have that extra 'trick' essential to a 1.75 rating puzzle ...

Is that a new requirement? As far as I'm concerned a 1.75 rating requires either an extra 'trick' or a lot of very difficult work. Am I wrong to include the second part of that? When I solved a puzzle that took me far longer than any other that I'd managed to solve I rated it as a 1.75 and felt justified in doing so.

Andrew wrote:Just a comment on one point

sudokuEd wrote:If the humans are agreed that this puzzle doesn't have that extra 'trick' essential to a 1.75 rating puzzle ...

Is that a new requirement? As far as I'm concerned a 1.75 rating requires either an extra 'trick' or a lot of very difficult work. Am I wrong to include the second part of that?

The trick not really a requirement, it's just something i notice in my walk-throughs of puzzles that are rated 1.75 by me. They always have something extra. It doesn't need to be a "trick"(like my Transformer puzzle trick really), it could be some really heavy combination analysis, really hard use of 45-tests or a very difficult single elimination that is needed to solve the puzzle. But they tend to have that something "extra". Of course it doesn't depend on that step alone. But it shows something for the difficulty that i need to resort to certain steps/moves. But as i say it is not a requirement, just something i notice in my walk-throughs.

greetings

Para