XC Debates: Top Girls' Team Ever

Though winter track can keep HS sports running enthusiasts somewhat preoccupied and generally far enough away from activities leading to a shorter life span, they do sometimes slide into discussions of dangerous topics such as the relative quality of the runners from past times versus the stars of today. The knock-down, drag-out free-for-alls online don't lead to much getting of wisdom since nothing can be proved and a lot of details are forgotten in the hazy recesses of time and our memories, but they're fun enough.

A topic that only goes a bit over 6 years into the past is the question about which HS girls' team should be acclaimed as the best ever. The issue had been raging on sites such as the LetsRun forum, and Bill Meylan has provided a great Speed Rating based look at the controversy over on Tully Runners using the top 3 teams from each of the 7 Nike Cross Nationals as a base of comparison. And though there's probably no need for more views to fuzz up the picture, I do have two cents left over from the holiday purchases to throw in, so I'll get rid of it. And since the only serious contenders are NYS teams (with apologies to a slightly off-balance Saugus\Newhall fan who posts elsewhere), the topic fits well here.

For years the Saratoga 2004 team had been crowned as the best team ever. Nicole Blood, Lindsey Ferguson, Hannah Davidson, Caitlin Lane, and Karyn Delay had a legendary year that saw them score 39 points in wins in the Manhattan-Eastern race, a then-record 18 in the States AA race, 22 at Feds, 51 at NXN, and also place 4 runners in the FLN Nationals, though after the rigors of the schedule none managed to place in the top 20 on the Balboa Park race course while Warwick's Aislinn Ryan got the win. When certain team issues limited Saratoga's record the next year in 2005 after two of the top quintet left the team (and Delay graduated in June 2005), it looked like no team would be able to touch the 'Toga 2004 squad on the podium of greatness for a while.

But a while goes by quickly these days, and last year after the veteran Fayetteville-Manlius scorched a perfect 15 at States and won their 4th national championship, the debate about who was best kicked up. The FM 2009 team included 3 seniors -- Hannah Luber, Mackenzie Carter, Molly Malone -- plus 2-time State champ Courtney Chapman, Katie Sischo, Jillian Fanning, and Heather Martin, and unlike the Saratoga 2004 team's fixed 5, the first 6 FM runners rotated places somewhat in the big meets. The team limited its diet to 22 points in the Manhattan-Eastern in a race that included national powers Saratoga, Queensbury, and Tatnall, The 74 points that FM racked up at NXN Nationals seemed to put them a notch below the Saratoga 2004 team, but the level of competition at the 2009 meet was very strong, as not only the familiar NY giants of Saratoga (runnerup), Queensbury, and Shenendehowa were running, but the field also included Ft. Collins, Saugus\Newhall, Tatnall\Wilmington and a number of other very powerful teams. Some of us felt that both teams had strong credentials to be the all-time queen of the trails, while I think a few others were not yet ready to let anyone share the throne with Saratoga 2004.

And so FM had to of course go one step further this year and make the debate even more heated. A squad that again included Chapman, Sischo, Fanning, and Martin, added breakout top runner Christie Rutledge and Katie Brislin and laid down more credentials that have never been topped. The 2010 team's 32 points at Manhattan-Eastern was not up to the 2009 team, and their perfect 15 in the States-AA race was perhaps not quite as perfect as the 1-5 overall finish the year before, but no team has ever dominated the NXN Nationals in its 7-year span as FM did in 2010. Their 27 point total at NXN was about half of Saratoga 2004's 51 points, and when you factor in the addition of the two more strong at-large teams that started in 2007, FM 2010's achievement boggles the brain cells even more.

So who really is the best? Opinion on Let'sRun is split, though probably shading more toward FM 2010 following the run at Portland. There is at least one 'Toga supporter who claims the 2004 team would have dominated this year's FM team, however. Tully Runner analysis gives a slight edge of 12 points to Saratoga 2004 in a simulated race that included the top 3 teams for the 7 championships using speed ratings that Bill created from the races weighed somewhat against how they related to season results to toss out unrepresentative outliers. It should be noted that the smallest margin of victory at NXN is 27 points by Hilton in 2005 (a race that one team mother stated that Hilton had dominated) and the average is 65, so an 11 point margin would be a virtual tossup for projections going into a race that would feature many runners competing against 3 or 4 of their younger or older selves. The simulations have FM 2009 a bigger 42 points in back of the 2nd pick FM 2010 team, just 2 points ahead of the FM 2008 team. The simulations point to 2004 as having the strongest set of top-3 teams. But as noted in the summary section, speed ratings from the season have some problems with predicting NXN performance, which is a very different animal from the NY State meet or Feds.

I have no real argument with anyone who pushes one of the above-mentioned top 3 teams, Saratoga 2004, FM 2009, or FM 2010. None of the 3 would be registering a huge surprise if they were preeminent at virtual Portland. To further muddy the waters, however, I am going to throw in a slightly different type of results analysis that ends up with a different pick than Bill got for Tully Runners. So many caveats and nonstandardized situations have to be looked at when weighing merits of the teams, however, that I will readily admit that the factors I use are just one nub of a bigger bone of contention.

In the final summary section, I will touch on the NXN comparative competition question for two of the years, and also throw out a few numbers on that most controversial of all questions: Which state has sent the best group of teams to NXN, and which state has been the biggest bust according to Tully Ratings projections? The answers to both questions are easy to guess and are alluded to throughout the article.

Critical Factors and Overview

When you judge who was the best team ever, do you look at the whole season or just the NXN races? To be fair, you obviously have to look at the entire season's accomplishments for the teams, not just how they did in one race. But since I am by nature an unfair person and the job of comparing and grading every race for the candidates is just too big, I am limiting the comparison to just the NXN races, just as was mainly done for the Tully Runners analysis. However, as has been noted, all three of the main candidates had spotless credentials for the pre-NXN season, so I'm happy to call it a wash, though other debaters will turn to the FLN race or myriad factors to argue their case. I am also limiting the final pool of candidates to 5 for expediency.

Another question is are all NXN races created equal? Clearly not, as conditions change from muddy (2005) to hard and fast (2008) and many in between on a course that has been tailored differently during the years. The level of competition also changes somewhat year by year, though for the most part the teams are at least the top 10 best in the nation among the 20 to 22 appearing there. Pre-2007, however, that point is somewhat more debatable, especially for the partial travesty of 2006. The makeup of the field for the last 4 races has differed a bit from the first 3 because of the addition of 2 strong at-large teams (the New York factor) to grow the lineup to 22. For the last three races, at-large individual runners have been competing at NXN, and they need to be factored out of the team comparison factors just as they are excluded from the team scoring. And one more factor that is hard to gauge and relates mainly to the Saratoga 2004 team is that 4 of their runners went on to compete at the Footlocker Nationals for the individual championship, which runners from the NE could still do through 2006. Running back to back killer races a week apart with cross-country travel and practice disruptions added in is surmised by some experts to be a possible cause of poor results in the latter race, though I am unsure how it affects the first race, in this case the NXN.

Looking over the top teams for the 7 years of NXN for five top candidates, it appears that you have to focus on the champions alone. The closest of the runnerups was in 2005, and though the Saratoga team of that year posted a score that would have won in 2006 (but they weren't there in any case), that year's team was too weakened from the loss of Blood and Lane to be a real contender. Weighing other factors, the top 5 do appear to be the Saratoga 2004, FM 2007, FM 2008, FM 2009, and FM 2010 teams that appeared at the top 5 of the Tully simulations. A quick overview of the 7 years and 7 NYS champions and with an NY bias is as follows :

2004:  Saratoga's epic year produced a finish of 3, 6, 7, 12, 23 at NXN for a 51 point score. Smoky Hill\Sisu finished with 125 points and is considered the top nonchampion team in the Tully simulations.

2005:  Hilton was a bit of a surprise winner in a slogfest over a Saratoga team that had beaten it by 1 point in the NYS-AA race and had dominated the Feds race that was a major part of selecting the NXN representatives. Hilton gambled by skipping Feds to build for NXN, and it was selected and scored 85 points while beating Saratoga (112) and Carroll-TX (123) even though Carroll had by far the best average time and Hilton had only the 3rd best.

2006:  A very, very young FM team that had been far back at the Manhattan -Eastern race came on very, very strong at the end of the season to nip Hilton and Saratoga by a 67-71-73 score in the NYS-AA race, The three teams finished in the same order at Feds, which left Saratoga at home for the NXN due to a selection process for at-large spots that to keep the language polite here just did not work. The 2006 FM team got its first national championship with a 128 score over 2nd place Hilton, but is generally considered the least powerful of the championship teams.

2007:  FM was already into it's now customary roll as it swept to its 2nd NXN win with 83 points (8, 9, 10, 19, 37) to runnerup Saugus\Newhall's 171 in a year that introduced two more at-large selections (NY picks) to avoid another fiasco like the 2006 Saratoga snub. The NY schools proved the change to be the right one with Saratoga\Kinetic finishing 3rd, Burnt Hills 5th, Warwick 11th, and NE school Hanover-NH finishing 4th.

2008:  FM rolled deeper with its 3rd title with a 2nd best ever 66 point total to that time with top finishers Courtney Chapman in 3rd, Meaghan Anklin in 9th, Hannah Luber in 10th, and Mackenzie Carter in 11th. NY competition at NXN was even stronger as Saratoga\Kinetic was 4th, Queensbury\Adirondack was 5th, and Burnt Hills was 8th.

2009:  FM at 74 won easily by 73 points over runnerup Saratoga\Kinetic despite not placing anyone in the top 10. A top 5 placing between 11 and 19 sealed the deal.

2010:  FM outdid itself by posting a score of 27 at NXN with Christy Rutledge 1st, Katie Sischo 2nd, Jillian Fanning 4th, Heather Martin 8th, and Courtney Chapman 12th. Saratoga\Kinetic was 2nd with 104. Only one other NY team was selected (apparently the CA underachievers have more prerace glitter), and North Shore\Glen Head finished 9th.

Rating Explanation

Looking at the results for the 7 years, it appears reasonable to select the 5 teams with the lowest NXN scores for the 5 candidates for best team as they all have strong unbeaten credentials. Those 5 are Saratoga 2004 and Fayetteville-Manlius 2007-2010.

Though the lowest score helps to narrow the field to 5 teams, I'm using two other comparisons to judge which teams dominated their races most thoroughly. Admittedly, when a team has by far the lowest score ever at NXN, it would be reasonable to say it was probably the best. But there will be arguments that the team may not have faced the most challenging field that year. FM 2010 has the lowest score of all the teams, but I guess it could be argued that it has so demoralized other teams that the performance level sank to a bad level by 2010, though commonsense seems to argue the opposite, that the level of competition is rising incrementally year by year. But by score order, the starting order for the teams is:

FM 2010 (25), Saratoga 2004 (51), FM 2008 (66), FM 2009 (74), FM 2007 (83).

Both of the factors I use to evaluate the teams are based on the much-abused average time score. There are issues with using average times in team comparisons, since teams often leapfrog a place or two ahead of teams that have superior averages, but they are good at providing a general synopsis of a team's strength. Obviously there can be a team like Carroll 2005 that has two top flying runners that gave them a great average time without meaning a huge amount in place scoring. On average though, a team with a significantly better average time usually wins a race between the two. And though average times of teams at NXN are meaningless in comparisons across the years because of different race conditions, any team's average time can be compared with pretty strong relevance to the teams it actually raced against. If you look at the difference between the average time of the race's champion versus the other top teams in its race, you have a pretty fair idea of how well they dominated the field. To ameliorate the affect of having one or two extra strong runners-up in races, I decided to compare the averages from the top half of the field, which means 11 teams for the FM candidates and 10 teams for Saratoga. I am not sure whether the competition got harder starting in 2007 after the addition of 2 more strong teams, but there is no extra weighting allowance given to the FM teams.

As a second factor, I used another average time measure, the average of the top 25 runners from teams at NXN (no at-large), and also see how the 5 candidates stacked up against the elite runners in the race. Based on their scores, going in you would expect that FM 2010, Saratoga 2004, and possibly FM 2008 would have much better or about the same average as the elite 25 average, and that FM 2009 and FM 2007 would be a bit slower.

Summing it up, there is no easily computed racing factor that  has anything close to the predictive accuracy for assessing NXN races than the team average time.

Analysis Results

The table below shows the data for the 5 teams for the average time comparisons and rankings. The definitions of the labels are:

Avg. Time: Average time for top five runners for the team.

AvgDvTH: Average difference of the team's average time vs. average of the top half of the teams in the race (FM averages vs. 10 other teams and Saratoga average vs. 9 other teams).

Rank TH: Team's rank among the 5 teams for the AvgDvTH.

AvgT25: Average time of 25 runners from teams in the race.

AvgDvT25: Average difference of team's average time vs. the average of top 25 runners in race.

Rank T25: Team's rank among the 5 teams for AvgDvT25

The data shows the FM 2010 team being the clear leader, and Saratoga 2004 being the clear 2nd best team. The rankings for FM 2009 and FM 2008 are less set, and FM 2007 occupies the 5th spot.

Again, this is only another way of judging the best team based on its performance at the NXN alone, but it does back up the top position of FM 2010 based on meet score, where it was far ahead of the other squads. Both FM 2010 and Saratoga 2004 are being only evaluated against the teams they ran against in one race, and not against each other directly, so everything is speculative.  Another factor though that would help FM 2010 here is to note that their 6th and 7th runners were far stronger than those spots for Saratoga 2004, so FM had a greater safety margin.

The biggest question is whether the competition level at the 2010 race was very different from the level at the 2004 race. The Tully Runners simulation hints at that possibility, and that's what will be looked at in the final summary section, along with a certain states' representation question that throws a little tarnish on the Golden State and pumps the imperial dynastic ambitions of the Empire State.

Summing It Up

So what can be said about the relative competition level between the teams at the 2004 and 2010 NXNs, and if there is a difference, does it strongly affect the evaluation of the results from the average time stats?

Admittedly, I don't have a clue as to how to accurately gauge how competitive the teams are that go to NXN in any direct way. But for each year, Tully Runners has been doing prerace projections for the NXNs based on the season's speed ratings (SR), calculated as best as possible. Since that's the only tool I know of, that's what will be used.

There are a lot of issues with speed rating projections when applied to the NXNs, however, so this evaluation is fuzzy at best. I'm sure Bill Meylan could list a lot of frustrations involved with compiling the national speed ratings that go well beyond the usual ones for the NY and NE region. Generally, when you start to apply SRs to schools that have little or no contact with the NY schools, the accuracy of the calculations are diminished as the base of comparison shrinks to almost nothing. It seems to be especially an issue with the CA schools judging by the NXN projections and results, though that may be also a kind of "state of mind" issue with the CA schools that compete there. In addition, as Bill has mentioned, sometimes the projected SRs for a team are way off because he does not know enough about the team to decide which of their races to evaluate, and sometimes the situation is pretty much hopeless as I discovered this year when trying to assess the Great Oaks-CA team for the Manhattan-Eastern race when the team had not run a 5K race yet.

Another issue with speed ratings that affects their accuracy for NXNs is that the meet is the Bermuda Triangle of XC meets, the place where teams go to watch their performances dive into the deeps. The motivation and running conditions that help teams build strong speed ratings up through their state championships and NXN qualifiers seems to disappear when they board the plane to Portland. The energy at races may also be strongly affected if like this year you are running against a team that has won the last 4 NXNs. Some of the stronger teams seem to kind of pack it in if they are not really running for a championship or to qualify for one. This year the 22 teams ran an average of 12 points below the team average that TR had projected for them, amounting to more than a half minute below expected performance for each runner. And 2010 was not an anomaly, 2004 runners ran better but were an 8 points below their expected level. A team called the Harford Harriers finished with a SR of 95 back in 2004, one of the all-time lows. It's not as though Bill can ratchet up the scores with an "NXN mental bog race condition" factor either, because there are runners like Christie Rutledge and Katie Sischo that go bust the curve and make the runners on other teams look like slackers.

Given two sets of data, the prerace SR predictions and the meet's actual team average times, which is the better predictor of NXN results? Note that for the NY State Championships, the difference between the two methods is very small and both are very accurate, though Bill's system does occasionally get snookered by teams that appear to be more interested in tying shoelaces than hanging up a championship performance (but it is a prerace estimate after all). For NXN races though, the difference in performance for the two methods is huge. Looking at the top 5 teams for the 7 races, the average team time placing corresponds to the team score place for more than half of the 35 teams, and the average difference of the predicted from actual is only 0.55. The SRs nailed a very respectable 10 picks for the 35 teams, and 13 other picks were only 1 or 2 places off, but 9 of the picks were off by margins ranging from 5 to 16, and the average discrepancy of 2.9 is more than 5 times higher than the average team time margin. The deviation total for 2010 alone is higher (20) for the SRs than for all 7 years of the average team time method (19).

Looking at the prerace SRs and results of the NXNs in 2004 and 2010 reveals a stunning fact. Of the 42 teams in the two races, 40 ran below their projected SRs. One team ran at their predicted level, and simply staying in place in performance helped Fremont-UT-2004 move from 18th in prerace projections to 5th, the 2nd biggest jump ever. Only one team actually did better than projections (2 points), and that was a very special team, the Stotans of 2010 (also known as FM). Saratoga in 2004 actually had a fairly sizable 6 point drop-off in the SR at NXN, and though it was expected to win by more than 100 points, the margin became somewhat closer when Smoky Hill-CO ran just a shade under their usual performance.

However, as computed by Tully Runners the level of the competition at the race was larger by an average of about 3.6 SR points per team for 2004 over 2010, and the results SR difference ballooned to about 7.5 points when the 2010 teams mainly went into a tailspin at the meet. If the SRs are accurate for prerace projections for NXN teams from across the nation, it does appear that FM 2010 was running against a much inferior field than the one Saratoga 2004 battled. Even if you do accept that very debatable point, you still have to ask if the 7.5 point difference in the race quality performance of the two fields was enough to justify moving Saratoga 2004 ahead of FM 2010 when the latter had far superior scoring and team time average difference results? Some will say yes, others will say no, and I'll just say that it's an interesting question but that the problems with the NXN SR data are too wide for me to be convinced either way. But if two teams finish in a tie in an NXN simulation and their 6th and 7th runners all took a wrong turn and headed south to a race in San Diego, I would pick as a tie-breaker a nod to the team that upped its performance for the NXNs, FM 2010. But Saratoga 2004 no sooner caught their collective breaths than they were mainly all off to run at Balboa Park in the FLN, so they certainly had an excuse if they left a little bit in the tank during an easy win.

Added note: using the NXN results SRs for the two teams, if you plug them into each other's races (expanding the field to 23 teams for 2010 and 21 for 2004), Saratoga 2004 wins the 2010 race by 2 or 3 points and the 2004 race by 7. This approximates the Tully Runner simulations for a field of the top 21 NXN teams. But again they both were only running against their own field of teams in their races.

Final word on the "Who's Best" debate is that FM 2010 vs. Saratoga 2004 would be a race for the ages, and that the only HS girls team that would stand a strong chance of beating them would be FM 2011.

On the question of states' bragging rights at NXN, there is really no question in a battle that pits the New York Titans vs. TheOtherStates Lilliputians. But a quick review of the 7 years' results and the TR prerace projections can be an enjoyable read for NY running enthusiasts. Four states appear to have sent 10 or more teams to the seven NXNs, the table below shows the performance of the state's teams for championships, teams finishing in top 5, teams finishing in the top half, the average prerace Tully Runners place prediction average for all teams, the average meet result for all teams, and the average difference between the prediction and the result to determine whether the teams exceeded or fell short of expectations.

New York is of course the dominant state, and considering that the NY teams don't have much chance to exceed the Tully Runner expectations either at all or by very much, that 0.70 score for average difference is exceptional. FM for instance has had an average pick of 1.2 for its 5 NXN trips and has a 0.2 difference. California teams, as noted previously, have been mainly big disappointments, as 11 of the 20 teams have finished 5 spots or more lower than projected, and 4 have finished more than 10 places lower. Only Saugus\Newhall and Corona Del Mar have performed well, and Saugus's recent spate of 4 top-4 finishes was preceded by an 18th place finish in 2006 when they were projected to finish 2nd. Except for two 3rd place finishes by Southlake Carroll (one off of a 20th place TR projection that Bill noted was based on missing data), Texas teams appear to have two guaranteed spots each year at NXN to finish around 20th. Colorado has had that strong 2004 runnerup Smoky Hill team, and for the last 4 years Fort Collins has bedeviled the TR projections by jumping up an average 4.5 spots and even taking 3rd in 2009.

Note that the selection of the 4 at-large teams for the girls NXNs seems to have been snake-bit from the start for teams outside of NY-NE. The 10 NY at-large teams and the 1 non-NY NE at-large have done well overall, with best-to-least results of (1, 2, 3, 4, 5, 7, 8, 8, 9, 11, 13), an average of 6.5 for place. CA's 6 at-large selections have averaged a 15th place, and the other states's at-large selections have generally finished way back except for the Fremont-UT underdogs in 2004 and the Mountain Brook-AL \ Caution team from 2007. Perhaps understandably, Nike tends to lurch around from year to year rewarding the previous year's overachievers (though it took a while for the NY girls to get the same status for bids as CA), only to get poor results. The allotment of 4 spots to TX teams in 2008 so that they could finish 13, 18, 20, 22 will always be a mystery, or maybe a result of where some darts landed.

So assuming the usual doings for NXN in 2011, the projections for the 4 or so CA teams that are selected next year will need to have a decidely negative Morning Line fudge factor added for the non-Saugus CA teams, while Fort Collins will need a little more love.