Automation Challenge: Experimenting With Scoring

The lastAutomation Challenge had some very small differences in the rankings, as well as some pretty extreme extremes (looking at you Dadvan...), so I was curious if a different scoring system might produce interesting results. It seems a bit unfair that a .1 difference in comfort could cost @EssExTee 13 points or that the @CarsOfFortLangley's abysmal comfort score only hit him for 5 points despite @HFV's 2nd to last score being almost 15 times higher.
To correct these "injustices", I tried out two systems based around the idea of continuous scoring. I normalized the scores in each category to a range 09 (we had 10 entries so it was easiest to start with ten discrete elements), based on the lowest and highest scores in each category, and then assigned points using a continuous function, so that Automation stats that were close together would get similar numbers of points. @TaylorMartin's stats were skipped for the normalization, so in Price where he came in first, his car actually received more than the maximum points (I kept track of his scoring as well). In the second method I did the above, but also added discrete bonuses for the podium finishers, so that there was still some extra bonus for finishing at the top. I describe both methods in more depth below.
Long story short though, the experimental scoring methods made surprisingly little difference for the minivans. Both approaches had me fall behind @t0ast (so obviously they are terrible ideas), and there was a bit of movement in the lower half, but the list more or less stayed the same. The Dadvan went nowhere.
Method One
For the first attempt I plotted the scoring and then computed a good best fit line to obtain my continuous function. A 6thorder polynomial seemed to produce a good one:
Except I was too lazy to type in all those terms, so I did a 3rd order instead (good enough for this test I think):
Now with my equation for pointing things out 09 I just needed to map the results onto that range.
Easy enough to do, just subtract the lowest score and then divide by the highest minus the lowest. Does get a little messier when you account for excluding @TaylorMartin's car:=(B3QUERY(A$3:B, "SELECT MIN(B) WHERE NOT A='"&INDIRECT("Sheet1!$A$4")&"' LABEL MIN(B) ''"))/(QUERY(A$3:B, "SELECT MAX(B) WHERE NOT A='"&INDIRECT("Sheet1!$A$4")&"' LABEL MAX(B) ''")QUERY(A$3:B, "SELECT MIN(B) WHERE NOT A='"&INDIRECT("Sheet1!$A$4")&"' LABEL MIN(B) ''"))*9
Rankings for categories that are supposed to be low are of course backwards, so I just changed the normalization function to be subtracted from 9, which flips them.
Once I have those numbers, it's easy to just plug them into the equation to get the score:
=0.0936*POWER(G3,3)0.729*POWER(G3,2)+(6.2349*G3)
Which leads to some changes. For example consider Safety:
Here's @TaylorMartin's scoring:
We have a big lead by @LongtimeLurker while @HFV, @Jawzx2, and @Brickman are separated by less than 1.5 points at the bottom.
With this new scoring method though, things change significantly:
32 points separate the highest and lowest entry, and @LongtimeLurker van (more of a bus really) is ahead of the nearest competitor by more than half of that range, so they get a massive points advantage to go with it. Part of that is due to the fact that everyone else missed the part of the curve the increases more quickly, so no one else got the builtin "highscore" bonus. Meanwhile the tight clusters in the middle and bottom all have fairly small number differences between people without much air between them.Same story with Comfort:
Here @EssExTee's near identical comfort score nets him near identical points. At the other end, @HFV only got a tiny boost over @CarsOfFortLangley in the official scoring, but under this method the points more accurately reflect that the Dadvan is really not a nice place to sit, even compared to cars of mediocre comfort.In the end though if we compare the totals:
We see that despite some heavy swings, the order ends up more or less where we were before. The RoadStar got a huge advantage in Safety, but its score actually goes down, largely due to falling behind the top choice in Service Cost and getting a score much closer to the competition.Method 2
There is a potential problem with the method above though, depending on your philosophy. While having closeness between close scores seems desirable, it may not be rewarding enough for our category winners and podium finishers. Perhaps instead of a continuous curve that gets steeper to reward the best scores, we should have a linear progression with bonuses for the finishers?
We can use the same normalization as above for this, but the scoring function is different. We start by taking the very simple linear equation:
y = 5x
Then we add 20 points for first place, 12 for second, and 5 for third. As with the previous method, this means that if our Automation scores were evenly distributed, we'd end up with the same results as the official scoring. The calculation does become a bit uglier to look at due to the need to filter out @TaylorMartin's results (for this I gave him the same podium bonus as the person after him):
=G3*5+IF(COUNTIFS(A:A,"<>"&Sheet1!$A$4,G:G,">"&G3)+1=1,20,0)+IF(COUNTIFS(A:A,"<>"&Sheet1!$A$4,G:G,">"&G3)+1=2,12,0)+IF(COUNTIFS(A:A,"<>"&Sheet1!$A$4,G:G,">"&G3)+1=3,5,0)
Obviously this method makes little difference at the bottom, but it can make for big changes at the top (the results are less of a change from the official results). Again considering Safety, with this calculation we get:
The RoadStar still gets a big advantage due to its dominating edge in safety, but @Zenith2455's StarCruiser and my Convergence don't lose out quite as badly due to the podium bonuses.
Looking at Comfort:
While with my other new system, I have basically no lead over @EssExTee, here he gets punished for finishing second (though also gets a bonus himself compared to lower scorers). The people in the middle take a bigger hit for placing out of the podium, while the bottom five are more or less on the linear section of both new scoring systems and so don't change much between them.The final results with the bonus system are even less changed than the other attempt:
All that happened was that @t0ast and I switched places, as did @Brickman and @Zenith2455 (skipping over @Jawzx2)Conclusion
Of the three systems I think my first attempt feels best, but ultimate didn't make much of a difference (at least with these cars), so I don't know if a switch would be at all worth while. It does have the nice feature that scoring remains the same regardless of how many players we have (though it will shift with high and low scores). Still someone who likes the podium bringing larger rewards might prefer the original or my second system, and it's easy to imagine that the fractional points seems needlessly complicated to some.
In any event, this was a fun little refresher on my spreadsheet skills, and I'd be happy to hear any thoughts from the @Automation crowd, or share results for other categories if anyone is interested. I'm also happy to try any other score function if you have something interesting, it just needs to map onto that 09 range.
Also @TaylorMartin, is the sheet you sent me how you actually do the scoring? I'm pretty sure I could whip up a version that takes the car names, the scores from Automation, and a few setting things and computes all the formatting, points, and tables. Let me know if that would be useful.

@facw I like the system where I get more points

@facw I've got a massive migraine right now, but I think your method 2 make sense to me as it's pretty close to the method I use on my own cars when I'm deciding what to enter. maybe instead of a set point advantage for podium it coud be a percentage? either way I agree that score that are close to gether should get similar points amounts.

@facw
I suspect the key problem with a points system is that noone apart from car reviewers actually uses one and even they don't understand it...and they pick something else to actually buy. Like the rest of us...So why should the Automation challenge scoring system make actual sense as opposed to any other sort of sense?

@facw good to see my van found new ways to come in last lol!

@facw No matter how many times I read this, math is like a foreign language to me. With that said, holy shit your scoring system is better!
is the sheet you sent me how you actually do the scoring?
Yep. This is how I've done it since round one. And it might be time for a refresh. If you want to hop in a discord call sometime next weekend to help me figure out how to do this then I'd be glad to implement it going forward. I'm a fan of your first method too.
@SilentbutnotreallyDeadly So why should the Automation challenge scoring system make actual sense as opposed to any other sort of sense?
The current scoring system barely makes sense to me. It also fluctuates depending on how many cars are entered. Awarding similar points to similar scores would be less devastating for those who finish a place or two below someone.
And if we try the new system and everyone hates it, then we go back to basics lol.

@taylormartin Cool let me throw together a nicer template (the sheet I used last night was sort of cobbled together with more automation as I got tired of doing things manually) and send it over, and if you have questions we can do a call to explain.
Pictured: Me

@facw is this comic written in a foreign language?

@facw @TaylorMartin @automation
Every time I try to write my thoughts on this scoring system something else comes into my mind and changes everything. But one thing I have concluded on are two or three factors at play now, that punish people unwarrantedly for coming in to a very close finishing position in events that are not decisive events, and overall proper weighting of categories given the mission of the vehicles.
I am in no way being personally critical of people or calling out publically how wrong individuals are. Oh no. So here are the factors I'm concerned with and some suggested solutions to run with to see how past events may come out differently or the same.
1.) weighting of the scores.
I think every category should be concerned regardless of the contest mission, as long as each category is weighted as a percent of importance. For example, if miles per gallon are not at all required and .5 mpg is acceptable to any buyer, than 0% of that score should be realized in the totals. But since .5mpg is wholly not realistic and any luxury buyer would actually have a problem with this, but 10mpg IS okay, than perhaps a 25% weight should be put on that score for the system total so it would matter somewhat, but not make a big difference overall. Now if someone were to game this by making a 100mpg supercar, it would notably cut its own throat in other categories enough to not gain any advantage here. Conversely if luxury were more important than anything, maybe make it 125% weighted score values towards the whole. This way when all things are considered the type of vehicle submitted is compared to the mission weightings fairly. So we don't have a car and driver bullcrap method of scoring a 911 critically low on trunk space when the buyers of a 911 don't care about that. However, they do need to fit golf clubs....so it's a teeny bit important. So about that scoring.... scoring
I do not believe a ten event scoring event should reward a single event last place finisher a 0 points total. If this were a single gold medal 100m final event, I can see last place getting last place but that should be for the FINAL TOTALS event  in a way much like how a Decathalon is scored. Where the total points are added up, much like the olympic Decathalon. So lets look at the last event and subject an alternate scoring method.
here is the safety category of the last event.
we can see @Brickman got zero points for a car that was not unsafe at all, just not AS safe as the winner. He was 62.43% as safe as the winner here; however, when whole categories are added together, why should he get a 0 for this absolutely punishing him for at least getting a D average in the class? If after 10 events his points total was last place, then yes he gets nothing. Conversely, let's also look at the middle seven cars that all scored in the 60 point region. @Zenith2455 through to @EssExTee, they scored from 67.8 to 61. Zenith scored 79% of what @LongtimeLurker scored and EssEx scored 71% of what Lurker scored, but in the scored awarded we see EssEx getting 15 points vs. Zenith's 52, effectively giving him only 23% of a value for running a 71% competitive car. Again, overall this isn't rewarding the whole value of the entrant.Here is my scoring of the safety event.
Safety (I suck at spreadsheets)
name  score  % out of 100
@lurker 85.7 100
@zenith  67.8  79.11
@facw  64.7  75.5
@t0ast  63.2 73.75
@grind 62.5 72.93
@cofl  62 72.35
@me 62 72.35
@sxt 61 71.18
@hfv 54.9  64.06
@jawz 54.7 63.83
@brickman 53.5  62.43okay. @LongtimeLurker set the max score of 85.7, since that is the 100% mark, everyone's scores here would be a % of that %. Get it? so the next seven cars that all score in the 60s would be given % points out of 100 based on that maximum score. So @LongtimeLurker lurker would get 100 points. and @Zenith2455 would get 79.11 points. see how this is working?
So when we set up the scoring of the categories that reward participation and effort that scores high, I think close competition should be rewarded equally and clear winners should also be rewarded equally. Poor performance would also be rewarded properly, for example, if someone was a totally unsafe car, and got 20% of the category winner, well then they deserve the 20% score! but they shouldn't get a 0 for being at least half as safe.
Let's look at the utility scores as well.
The top utility score was a 48.3, and that was pretty hefty. But the lowest score in this minivan/people mover event was a 4.1. Was a 0 deserved? well in this case the % of that high score was so low it was 8.5%, so nearly a zero awarded here but there was at least SOME utility brought to the table.In conclusion, I think the zero points given is not appropriate if there was a decent vehicle still produced. To give an example, in a heavily competitive event where the top winner brings an 80 point car, and the loser brings a 70 point car in that category, and lets also make this a weighted double points category, should that 70 point car get a fat 0 and be totally crippled overall where it actually performs well in other categories? No. I don't think so.
So I am all for weighting categories according to the mission event, as percents of importance where 100% is equal importance, and then let the final scores added dictate overall placings. We don't need algorithms or long strings of formulae for this.
I hope I'm not building a mud puddle here. Or calling out efforts as incorrect. This is just my thinking on the subject and something that would make a fairer scoring rewarding close competition. and punishing a crap result in comparison. Because when a manufacturer brings a game changer, well the scores of the lesser cars would indeed be much poorer, but not a 0.
Final note on bonus points for beating @TaylorMartin ! If the bonus points are 5, well then that 5 should go to the top finisher in that category (as 100% of that bonus) and a percentage of it down to 0 to taylor based on how far above him you scored. (also, on his score tables, that 5 points equals 7.7% of a bonus score on my scoring system...)
Thats it. run with it if you'd like. I'm just poor on spreadsheets or id try to throw the last event together based on this method.
thanks for your time.

@grindintosecond said in Automation Challenge: Experimenting With Scoring:
Here is my scoring of the safety event.
Safety (I suck at spreadsheets)
name  score  % out of 100
@lurker 85.7 100
@zenith  67.8  79.11
@facw  64.7  75.5
@t0ast  63.2 73.75
@grind 62.5 72.93
@cofl  62 72.35
@me 62 72.35
@sxt 61 71.18
@hfv 54.9  64.06
@jawz 54.7 63.83
@brickman 53.5  62.43
okay. @LongtimeLurker set the max score of 85.7, since that is the 100% mark, everyone's scores here would be a % of that %. Get it? so the next seven cars that all score in the 60s would be given % points out of 100 based on that maximum score. So @LongtimeLurker lurker would get 100 points. and @Zenith2455 would get 79.11 points. see how this is working?I really like this method... I always tried to make sure the zero mark didn't make or break a car by limiting just how far ahead the frontrunners could get. But this is better (and at the very least, I'm able to understand it... it just requires a hell of a lot more math).
I will say that scoring is one of the aspects of these challenges that takes the longest, right after putting the whole thing together. I wanted to make the math as simple/copypasteable as possible so that I could get it done faster. But this month will definitely see a different scoring system.
I think every category should be concerned regardless of the contest mission, as long as each category is weighted as a percent of importance.
The only reason I disagree with this is that there are a lot of categories. The more categories there are, the longer it takes to score all of them.
Final note on bonus points for beating @TaylorMartin ! If the bonus points are 5, well then that 5 should go to the top finisher in that category (as 100% of that bonus) and a percentage of it down to 0 to taylor based on how far above him you scored. (also, on his score tables, that 5 points equals 7.7% of a bonus score on my scoring system...)
I'm sure this is a great explanation, but I don't understand it. Elaborate just a little more.

@taylormartin @Grindintosecond, I did consider whether it would be better to just map to percentage of the total score, rather than percentage of the score range, but for my experiment I wanted to keep it more similar to the existing system. It wouldn't be hard to do the computations that way. Makes computing the mapping a little more complicated for categories where low is good, but otherwise it is just changing (scoremin)/(maxmin) to score/max. I'm worried it might end up with everyone scoring more or less the same in most categories, whereas clamping it to the minmax range guarantees some variation, and makes it more uniform between categories.
I don't think scoring every category (with weights) is a good idea simply because I don't think we want @TaylorMartin to have to transfer the scores for every category. I do think it might be useful to have weights. I'm going to add that into the template I'm making.

@taylormartin sure. So the math isnt so bad really.
Take the top score and that is your 100% mark. every score below that is divided by it. Zenith's 67.8 / lurkers 85.7 = 79.11% of his score. so that's the points for the event. Then weight the category by desired importance and their points get neutered or magnified by the % assigned. supersafe car event? well 150% assigned so lurker gets 150 points, and Zenith gets 118.67...and brickman 93.64, and not 0x1.5 for a score.
Now your bonus
can be a bit complicated but not so bad really just adjusting numbers as 0 or as 100%.
Bonus could just be another 7.5% added to the top (107% now possible should you keep giving a bonus) score and then the position of your car (that 62) would be the 0% payout. making only 23.7 points the variable from 0 to 100. So each point now equals 4.219% of that 7.5....now its sounding convuleted.,
here is the math that seems to explain it.
winner  Taylor's score = 23.7 (or X)
.075 (the fixed bonus%) / X = bonus percent score per point above Taylor.so in practice
.075 / 23.7 = .003165 per point above you.
or, each point above you to the category winner will net .316% point score. so the category winner lurker will get +7.5 and zenith would get +1.835. This rewards someone the most bonus for totally smashing it, or awards the top smashers should they be close.this spreads that bonus 100% from your position (0%) to the category winner.
category score x weighted corection=category final score. add all categories together = total score. then final positions are seen and medals awarded and parties on corporate yachts commence.

@grindintosecond said in Automation Challenge: Experimenting With Scoring:
or, each point above you to the category winner will net .316% point score. so the category winner lurker will get +7.5 and zenith would get +1.835. This rewards someone the most bonus for totally smashing it, or awards the top smashers should they be close.
I kind of like the idea of a "set" amount of bonus points for everyone who beat me, but that exponential bonus does kind of make sense... It's basically taking another percentage of the difference between the top score and my score.

@taylormartin said in Automation Challenge: Experimenting With Scoring:
@grindintosecond said in Automation Challenge: Experimenting With Scoring:
or, each point above you to the category winner will net .316% point score. so the category winner lurker will get +7.5 and zenith would get +1.835. This rewards someone the most bonus for totally smashing it, or awards the top smashers should they be close.
I kind of like the idea of a "set" amount of bonus points for everyone who beat me, but that exponential bonus does kind of make sense... It's basically taking another percentage of the difference between the top score and my score.
yeah, that's works well. Maybe the guy just above you gets 4, everyone else above gets 5 and the top guy gets 6? I dunno....its simpler that way I guess.
Ill probably open up Xcel and start on basic math stuff for fun and then see how past results work out.

@grindintosecond Here's what I meant in my message:
It looks like the first three boxes in section B have this issue. I'm sure it's an easy fix, I just don't know how to do it myself haha!
Looks like the drivability score is also affected. Otherwise, the sheet works as advertised which is really cool!

@taylormartin fixed the math issude , changing the second row columns now to make price and visuals in there.

@taylormartin OK, so I finally had a chance to make you a template (this uses my first new scoring method, but we can try some others if desired). I'll share it with you in a minutes.
Here's how it works (using the minivans as an example):
 Open the sheet and do File > Make a copy
 Name the copy (which is what you'll be working on) and take a look at it:
 Do the setup:
For each category being scored, replace "Category #" with the name of the category (It's fine if there are extra unused ones). Then set the ASC cell for each category to "ASC" (without quotes), and set the Weight cell to how strongly you want each to count (e.g. if you want a category to count double, set it to 2). Fractional weights are fine too. You should have something that looks like this:
(I set the last two to 0, but it shouldn't matter, they will be 0 regardless)  Enter in all the stats from Automation and the poll (make sure to put your car on the first line, otherwise order doesn't matter):
By default all the columns are going to display to one decimal place, so for money columns, you should should change the number format to "Currency (rounded)".  Now you can look at the category tabs, so click "Category 1" and you'll see this, with the scoring and results table generated from the input data:
 However, there's unfortunately one more manual step needed to get the over all results. You need to change the tab name to be correct for the category so that the results page can find these results to total. So change:
to
and repeat
It's easiest just to copy them from cell B1 on each category page, as it's important that the labels match.  Now the Results tab should have been populated:
Note that if some columns that should be populated are all 0.0, you can fix this by clicking on a cell in an empty column (like R1) and clicking backspace, which will force a recalculation.  And you'll see the final results table at T1 on the Results tab:
That's it, you're done! Except for all the writeups.
A few caveats and notes:
 This is set up for 30 players max. It's easy to add more, but it doesn't seem especially likely.
 It currently supports max 14 categories. We can add more, but a few references will need to be updated.
 I left you car in the scoring since we can score it with this method (though you lose out on the visual poll of course). If you'd rather not see it in the final table, I can add a WHERE clause to remove it.
 I skipped the 'X' for you in the Bonus column, because it was giving me some weird behavior with QUERY(), but if you really want it, I'm sure I can get it in there, though the red highlight seems enough.
 Scoring weights currently also impact bonuses (e.g. a weight of 2 will give a bonus of 10, while 0.5 will yield 2.5). This is easy to change if you think bonuses shouldn't be doubled.
 As above this scoring is my first experimental scoring method which I liked better than my second, yours, or @Grindintosecond's. But you are running the show here. If you want to see something different I can probably change the scoring function pretty easily.
 I'm still using the "rough" scoring function because something went wrong when I did the more complex one. I'm going to try again to see if we can get rid of that .3 on the high scores (caused by the function not being a perfect fit).
 If you find manually naming the tabs too annoying, I should be able to add a script that does it automatically, but I wanted to do this with only formulas to keep it simpler for nonprogrammers. Alternatively, if you are fine with the tabs just having generic "Category #" names, I can support that, but I found it really annoying not to have them have descriptive names.
In any event, give it a try (maybe try putting in the classic sports cars or something) and let me know what you think, if you have any questions, or if you see any bugs.

@facw I can't work on this sheet now, as I'm not home, but you and @Grindintosecond should collaborate on this sheet:
https://docs.google.com/spreadsheets/d/1kwFLGpZvtgb6ixmcScJFY7ueW4zHhaYDVjELneEyjDY/edit?usp=sharing
Grinds made this one and, from what I understand, the system you're using is fairly similar. His is a little less harsh on those lower down the line, which I kind of enjoy. It's just percentages of the winner's score, so if the winner scores 100 and the loser scores 50, that loser gets... well... 50 points.
I appreciate the input from both of you, seriously. The next challenge is sure to be the fairest one yet!

@taylormartin I'm going to try @Grindintosecond's method, but my initial impression is that it's not good to have that sort of scaling. When we have a minimum safety of 50 for the minivans, does it really make sense to give someone who did the minimum (or a tiny bit better) 50% of the available points? Also a strict 0100 isn't good both because it doesn't work at all for time or money categories, and even for things like driveability, you might have something like the classic cars where all the stats are terrible. To make it work you will have to manually set a min and max for the scoring range for each category.
I'm planning to add Min and Max settings for each category to my sheet. If set to MIN or MAX it will behave like my scoring with the top and bottom of the range determined by the lowest and highest scores. If you replace with a number though, it will substitute that as the min/max, so you can clamp the range if you want to.
I do like the idea of making it nonlinear, so that if you do score really well, you get more points per category point than someone who scored near the bottom (mimicking your linear with a bonus for podium features).

@facw said in Automation Challenge: Experimenting With Scoring:
Also a strict 0100 isn't good both because it doesn't work at all for time or money categories, and even for things like driveability, you might have something like the classic cars where all the stats are terrible.
I don't think that's what he did. If the person in first place scored 20 drivability points on their classic car, and the person in second scored five, the winner would get 100 and the loser 20, right?
Maybe?
Idk, Math is wonky.
I do agree on the minimum safety though. Perhaps for categories where I set a limit, the closer you are to th lower end of the spectrum, the less points you should get?

@facw
My whole point was to reward the delta variance. So if the minimum safety is 50, and someone brings 20 to the party, well it just won't score high at all in comparison to the winner's 70. So they will get a deservedly horrible score compared to everyone else. Now they have some safety, but its bad. so the score is bad. Did they bring a completely 0 safety to the game? That would score zero and would deserve a 0.So with another angle here. If someone doesn't meet the expectations of the minimum (50 safety in this case) for a category, it's up to the contest itself to grade separately and intervene, where the score entered could be a fat 0 and not the 20 points for not qualifying for that category and being dangerously unsafe, and then punishing, possibly deservedly, for the egregious offense of not meeting the contest rules minimums. So in fact they would see that 0......
So should the 0 be a punishment for not meeting the category ruleset minimums? We should seriously consider that. I may be in favor of it! this is safety! (or in the case of emissions minimums, 'this is the environment!')
Now, let's look at the other angle. If the minimum safety is 50 and the winner brought only 50, and everyone else brought 49, 48 respectively, then nobody is the absolutebyfar clear winner here, they have only just won, and everyone brought a more equal Agame to the winner. so they are rewarded with a near 9899% score because they did nearly just as good a job as the winner. We could split hairs over who was in first and second, but that isn't for the category, that is for the final tabulation placings. Just like the cumulative scores if any auto journalism magazine having a tenbest contest over many categories.
Would 060 times of a car deserve a terrible points award for 8th place if their 060 time was .08 seconds slower than the winner? I think not. That would be wholly unfair to the manufacturer for making something so very almost close. They might have more miles per gallon, comfort, AND lower maintenance costs and still lose the whole event because of that one 'tanked' score giving them 1/10th of the winner's score, especially if it's a highly weighted event. That is what I'm attempting to eliminate in the end results by using my 100% scale. A 4way tie separated by only decimals of scores in one event shouldn't doom that 4th person in the whole picture.
My opinion to consider. So I made a spreadsheet to see what would happen. I'm glad were discussing this to find a fair solution.

@grindintosecond I'm going to run some numbers tonight to see how various methods compare. I see you point, but I don't we've ever really had categories where the entire field was so clumped together that being a tiny bit off the leader rewards last place with nothing. Usually the last place seems like it's someone's dump stat, or people just did the minimum or there abouts, and in both those case, a 0 seems fitting. I'm concerned that with you approach, things may bunch up enough that many categories become basically meaningless (or meaningless if you don't win). But that's just intuition, I could be wrong. I'll post a whole bunch of numbers tonight to try to get an idea of what looks best.

@facw said in Automation Challenge: Experimenting With Scoring:
Also a strict 0100 isn't good both because it doesn't work at all for time or money categories, and even for things like driveability, you might have something like the classic cars where all the stats are terrible.
Oh yeah, I forgot to address this. How does it not work for all time and money categories? Whoever is the lowest priced, may win that by far, and tank other scores, but the case is, if that person didn't cut their own throat and throw away scores elsewhere in many other categories to get that score, then they did a VERY good job of total design. Is that not what we are rewarding?
And if everyone brings a horrible score, then the best gets something. and the worst is close too....and in one event, isn't that worth something? being really close? (ie: the 060 concept above)

@facw said in Automation Challenge: Experimenting With Scoring:
@grindintosecond I....... I see you point, but I don't we've ever really had categories where the entire field was so clumped together that being a tiny bit off the leader rewards last place with nothing.
Oh we had a 4 way near tie last event where the point spread went from 67 to 60 over , umm, 5 people? and other points awarded variance was nearly half for the 4th person in that group over what the first of them got.
Usually the last place seems like it's someone's dump stat, or people just did the minimum or thereabouts, and in both those case, a 0 seems fitting. I'm concerned that with your approach, things may bunch up enough that many categories become basically meaningless (or meaningless if you don't win). But that's just intuition, I could be wrong. I'll post a whole bunch of numbers tonight to try to get an idea of what looks best.
I hear that, but over the course of an entire contest, cars more equal would at least be next to each other in the final standings. I scored the old sports car event again and I think it was 5th and 6th were defined by 0.91 points. They were very equal over the whole course. But it wasn't a whole line of cars that close. First was 720 points, last was high 500's. and there was enough of a difference between the 10 entries. I even rescored the #2 event of the communist budget cars, and one car did dump the MPG stat and got 20% of the high. That hurt them and rightfully so. But if they dump, then okay. they get hurt. Perhaps a minimum score for every category to keep things more honest? Because if a category is lightly weighted than that would be a tactic to watch.

@grindintosecond @TaylorMartin OK, so I've expanded my sheet and run some more numbers. Here's a link to the new template:
https://docs.google.com/spreadsheets/d/14kB51ONtWCHke6FM2us33IXJZ3mWbzelEZcA_N3C4rg/edit?usp=sharingChanges include support for setting the min/max for a category (it can still pick based on the results), and easier way to specify the scoring function (if it can be expressed as a polynomial), running scores to 100 rather than 65 using the default scoring (I've left the "beat @TaylorMartin bonuses the same so they are a bit less potent compared to a 65 score). I also map results onto a 01 scale instead of 09 (because who normalizes to 09?), though that shouldn't change anything.
With a more flexible sheet, I rescored the minivans using a few different setups, and I'll show the results below (apologies to @Grindintosecond if I butchered your idea, had some stumbling blocks when setting it up, feel free to suggest improvements.) One idea worked out really poorly (the Dadvan nearly doubled everyone else!), so clearly that's not the way to do it, but I'll go through the other results.
I'll go through some categories to compare. I've scaled the official scoring by 100/65 in each just to make it more similar to these new methods.
Here's driveability:
The six new methods are combinations of three different ways of setting the boundaries and two different scoring functions. The first scoring function, (NL) that emulates @TaylorMartin's approach of having linear growth through the first ~2/3 of the range and then more exponential growth through the last third (i.e. bonus points for people who score near the top). The other (L) just multiplies the normalized value by 100, it is completely linear.The three ways of setting the range are:
 (Min/Max) which just takes the highest and lowest values (excluding @TaylorMartin's) and normalizes everything to that range.
 (0/MAX) is similar but instead of setting the min to the lowest score, it sets it to 0 (I believe this is what @Grindintosecond was proposing above?). It does get messy for the categories that don't fit neatly in a 0whatever range, especially if the category is one where lower values are good. There if you set 0 as the best option, all the scores will be very low, and there is no natural max, so the whole range is poorly defined).
 (30/70) is basically just to pick reasonable min and max values for each category (for example saying that a score of 30 is worthless and a score of 70 should be worth full credit). Put frankly this sucks. It's really hard to pick good values, and it would be even harder without looking at the numbers beforehand. I really wouldn't want to do this if I were @TaylorMartin. This is also the one that the Dadvan broke for reasons I'll explain below.
In any event drivability is a category that doesn't have much weird going on ,being reasonably well distributed with only the Dadvan as an outlier. All of the methods except the official scoring do a decent job keeping close scores for vans with similar drivability, though the 30/70 NL gives way too many points to the leader. This is because for values outside of the range they get ever more bonus points. Probably for it to work, at least with these values it would need to be sigmoid function or something, but I don't really think that's needed since frankly the 30/70 results all sort of suck. I'm going to ignore them here , for that to work, you need to select good values, and that seems too hard.
Moving along, the 0/MAX methods both give what seems like too many points to some vans with pretty crappy drivability. A car with 42 drivability really doesn't seem like it should get more than half of the available points. All of the new methods here provide significantly more points for the lower places as long as they didn't finish last, though I'm not sure if that's a good thing.
Let's talk safety since that's a place where there is a big difference between the methods:
This is a hard case. We have a limit so no one is going below 50. We also have one super outlier with a very high score. Personally I don't think it makes any sense to award such large point totals to cars that barely scraped by the minimum. Seems perfectly fair for that to be worth basically zero. We could work around that by setting the minimum instead of 0 here, but that basically works out to my method anyway. On the flip side, my method gives a huge edge to the RoadStar. Maybe that is deserved (the gap between first and second is bigger than the gap between second and last), but it is a big jump.@Grindintosecond, which one did you think everyone was all bunched up in? Was it safety? The middle is close together, and here the bunch gets similar scores, but different between methods. The 0/max methods are obviously much more generous to them than the min/max method, but my feeling is that the RoadStar probably does deserve a big lead here, for being so much better, while they don't deserve a lot of points when when they aren't much better than the minimum.
Also, another question for you @Grindintosecond, how did you see your system working for the "smaller is better" categories? Feels like it's either going to be similar to my system, work poorly, or require manual range picking? I won't go over those sorts of results here since I ended up doing basically the same as my system, but I'd be curious to know if you had a better way to map those.
In any event since I'm getting tired of writing this out, and most categories look roughly similar I'll just make my observations:
 I much prefer the mixmax mapping rather than 0max, which just seems to give too much credit to cars that really didn't do very well. Sure if every car is identical weird stuff happens, but that doesn't seem like a huge problem. Instead we just muddle categories by not letting cars that are good at things gain much ground in those categories.
 I do think in either method the outliers cause some weirdness. I like giving high scoring cars a bigger bonus for high scores, but in the minivans there are a lot of cases where everyone but one car miss the "ramp" and so they can pick up a really huge advantage. Maybe the linear scoring is better? Or linear with bonuses based on place?
 On the flip side, my feeling is that if you made a car that was way better than what everyone else did, you should get a big boost, since you probably had to make significant sacrifices or do something clever to get there. And if cars are bunched together it implies that doing better is hard, so a car that does better should see a reward, and even small gains in the bunch should warrant jumps. Of course that may be too gamey, if adding .2 sportiness is really hard, it doesn't necessarily mean that a car with .2 more sportiness would be noticeably better in the real world.
 If we went with something more like @Grindintosecond's we definitely need minimums other than 0 for columns that have requirements, because nothing interesting is happening below that point.
 Most importantly I think the cars are more fun to build when they are more distinctive. That means that we should encourage people to try for wins/podiums instead of just making a car that's a jack of all trades. I think nonlinear scoring helps with that, since it means that doing badly in some categories is less damaging when you can make it up with bonuses in others. I'm not sure if the 0/Max scoring works against that, but I do think the linear scoring does, and encourages unexceptional cars.
Anyway my preference between these four would be:
 Min/Max clamping with a nonlinear function (though maybe toned down a bit from what I have)
 Min/Max clamping with a linear function
 0/Max clamping with a nonlinear function
 0/Max clamping with a linear function (this one just seems way too forgiving to me)
I do think all the methods are an improvement over the strict rank based scoring (no offense @TaylorMartin)
In any event, I'd love to hear feedback, and of course @TaylorMartin, it's your show, so you certainly should just feel free to say you like a certain way of doing things.