Statistical Draft Modelling: Where the Models Have Missed

I know a lot of people here use my projection models to help inform their evaluations of prospects. I am generally happy with the performance of both the EWP and HUMBLE models, but there is no question that they sometimes make mistakes. The goal of this post is to pull up some of the more extreme historical cases where the models flubbed. Hopefully, some patterns will emerge that help us gauge the reliability of the model for different 2015 prospects.

I took all of the players drafted between 1990 and 2012, and subtracted their observed NBA wins peak from their projected average between EWP and HUMBLE models. At this level, the results are a bit awkward, since guy like Dirk, Paul, and Durant, all of whom the models loved, top the list of "underrated". To make sure I am focusing on guys who the models actually incorrectly assessed, I limit the "overrated" set to guys with an AVG expectation greater than 5 and the "underrated" set to guys with an AVG expectation of less than 5.

I will present these lists with some limited commentary, but I hope for some discussion of patterns in the misses in the comments.

Point Guards:

Overrated:

Name	Pick	Season	Pos	MP	EWP	HUM	AVG	OBS	DIF
Jay Williams	2	2002	1.0	3573	6.8	6.9	6.8	0.9	-5.9
Ricky Rubio	5	2009	1.2	3684	12.3	8.6	10.4	7.5	-2.9
Kyrie Irving	1	2011	1.2	303	11	9.9	10.4	8	-2.4
T.J. Ford	8	2003	1.0	2178	5.1	6.8	5.9	3.9	-2
Dion Waiters	4	2012	1.5	1462	3.6	7.4	5.5	3.8	-1.7

In general, the models do a good job of not becoming overly excited by point-guard prospects. Jay Williams tops the list, but he failed for a reason that was completely impossible to predict. Of the non-injury cases, both the top two are great players who simply haven't (yet) lived up to lofty model expectations.

Underrated:

Name	Pick	Season	Pos	MP	EWP	HUM	AVG	OBS	DIF
Steve Nash	15	1996	1.1	3402	2.2	3.2	2.7	12.3	9.6
Sam Cassell	24	1993	1.4	2344	2.4	3.1	2.8	9.5	6.7
Deron Williams	3	2005	1.1	3198	3.3	6	4.7	10	5.3
Terrell Brandon	11	1991	1.0	2175	3.2	4.3	3.8	9	5.2
Rafer Alston	39	1998	1.0	1030	1.7	2.3	2	6.6	4.6

Interestingly, scouts seem to do a good job of picking out great prospects who my models miss on. Both Steve Nash and Deron Williams were drafted well above where the numbers recommended and became stars. Some other cases including Westbrook, Wall, and Rose passed the 5 AVG cutoff but were still definitely underrated by the numbers as well.

Shooting Guards:

Overrated:

Name	Pick	Season	Pos	MP	EWP	HUM	AVG	OBS	DIF
Nick Calathes	45	2009	1.5	2641	7.2	3.9	5.5	2.5	-3
Todd Day	8	1992	2.3	2840	5.7	6.2	6	3.9	-2.1
Tyreke Evans	4	2009	2.0	1073	5.9	6.8	6.3	4.7	-1.6
Bonzi Wells	11	1998	2.2	3540	7.4	5.7	6.6	5.2	-1.4
Jim Jackson	4	1992	2.4	3165	4.5	6.5	5.5	4.1	-1.4

None of these are actually that bad. Calathes (who is probably more a PG) was clearly overrated by the EWP model, but the rest are pretty accurately pegged. This is great news for D'Angelo Russell, Emmanuel Mudiay, and Delon Wright are are projected as "shooting guards" (Pos 1.5-2.5) and score 10.6, 7, and 5.3 respectively.

Underrated:

Name	Pick	Season	Pos	MP	EWP	HUM	AVG	OBS	DIF
Brent Barry	15	1995	2.0	3123	2.3	4.8	3.5	9.2	5.7
Brandon Roy	6	2006	2.3	2838	2.6	5.6	4.1	9.7	5.6
Michael Redd	43	2000	2.0	3324	3	2.5	2.8	8	5.2
Cuttino Mobley	41	1998	1.7	2762	1.6	1.1	1.4	6.6	5.2
Lance Stephenson	40	2010	2.2	902	1.7	1.5	1.6	6.8	5.2

The model seems to have bigger issues in under rather than overrating shooting-guard prospects. The model completely missed on several excellent shooting-guards. Barry and Redd can both be explained by dramatically improved three-point shooting after college. Roy was flagged for poor steals and blocks which apparently wasn't an issue. I still don't know what to think about hyper-recruit and bland college player Lance Stephenson. Devin Booker and R.J. Hunter should be happy about this.

Small Forwards:

Overrated:

Name	Pick	Season	Pos	MP	EWP	HUM	AVG	OBS	DIF
DerMarr Johnson	6	2000	2.8	879	5.3	6.7	6	2.4	-3.6
Joe Smith	1	1995	3.4	2098	10.1	9.1	9.6	6.5	-3.1
Billy Owens	3	1991	3.2	2403	5.8	7.9	6.8	5.9	-0.9
Tracy Murray	18	1992	3.1	2949	5.5	5	5.2	4.4	-0.8
Toni Kukoc	29	1990	3.0	115	8.4	NA	8.4	7.7	-0.7

Basically DerMarr Johnson, is the only real error on this list. You might be thinking "Joe Smith isn't a 3" and you'd be right. Positions are assigned by a model projecting NBA position from NCAA stats. Joe Smith is apparently an error on that model as well. The model actually pegged the last three quite accurately.

Underrated:

Name	Pick	Season	Pos	MP	EWP	HUM	AVG	OBS	DIF
Gerald Wallace	25	2001	3.3	824	4.9	3.8	4.3	10.3	6
Bryon Russell	45	1993	3.3	2334	1.8	1.9	1.8	7.8	6
Joe Johnson	10	2001	2.6	1605	2.6	4.4	3.5	8.2	4.7
Jimmy Butler	30	2011	2.8	3033	2.9	2.8	2.8	7.3	4.5
Richard Jefferson	13	2001	2.9	2211	4.3	4.9	4.6	8.6	4

The good news is that even if you struggle by the models' standards, you could still end up an NBA star from the SF spot. The patterning of under/overrated is is great news for Winslow (8.9), Johnson (7.2), Oubre (5.9), Hezonja (5.2), and Rondae Hollis-Jefferson (5.2) (as well as Wiggins from last season and Otto from 2013). Small forwards seem much more likely to exceed rather than fall short of expectations.

Power Forwards:

Overrated:

Name	Pick	Season	Pos	MP	EWP	HUM	AVG	OBS	DIF
Tyrus Thomas	4	2006	4.0	829	12	9.1	10.6	4	-6.6
Michael Beasley	2	2008	3.7	1194	9.1	8.8	8.9	2.9	-6
Doug Smith	6	1991	3.7	1993	4.9	5.4	5.2	1	-4.2
Brandan Wright	8	2007	3.8	1014	6.4	6.9	6.7	3.5	-3.2
Stromile Swift	2	2000	4.1	1332	6	6.9	6.5	4	-2.5

Relative to the wing positions, it looks like 4s are more likely to bust. Tyrus Thomas and Michael Beasley were huge misses. Unfortunately, these were all cases where both the numbers and the scouts were mislead. That makes it difficult to sort out the busts in the future.

Underrated:

Name	Pick	Season	Pos	MP	EWP	HUM	AVG	OBS	DIF
Antonio Davis	45	1990	4.1	991	1.4	1.2	1.3	7.4	6.1
LaMarcus Aldridge	2	2006	4.4	1602	3.8	4.7	4.2	10.1	5.9
Paul Millsap	47	2006	4.0	3239	4.4	4.6	4.5	9.6	5.1
Carlos Boozer	34	2002	4.0	2621	4.3	3.9	4.1	9	4.9
David Lee '05	30	2005	3.9	3171	3.4	3.1	3.2	8	4.8

The model also struggles in the other direction with power forwards. The models actually did better than the scouts on Millsap, Boozer, and Lee... but the professional decision-makers were much smarter in their praise of LaMarcus Aldridge.

Centers:

Overrated:

Name	Pick	Season	Pos	MP	EWP	HUM	AVG	OBS	DIF
Greg Oden	1	2007	5.0	925	13.1	11.2	12.1	4.3	-7.8
Shawn Bradley	2	1993	5.0	984	12.7	8.9	10.8	7.5	-3.3
Bismack Biyombo	7	2011	4.6	239	6.2	8.7	7.4	5	-2.4
Andrea Bargnani	1	2006	4.5	2365	5	5.8	5.4	3.1	-2.3
Enes Kanter	3	2011	4.8	697	7	6.4	6.7	4.6	-2.1

I won't worry about missing on Oden. Bradley is an unsurprising wonky case, and given that he was actually pretty solid I won't worry about that one either. The bigger issue seems to be international big-man prospects. This could be a flag on Porzingis who I am already worried about.

Underrated:

Name	Pick	Season	Pos	MP	EWP	HUM	AVG	OBS	DIF
Brook Lopez	10	2008	5.0	1487	2.8	3.5	3.1	8.8	5.7
P.J. Brown	29	1992	4.6	2539	2.8	3.2	3	8.3	5.3
DeAndre Jordan	35	2008	4.9	781	3.1	2.2	2.7	7.7	5
Theo Ratliff	18	1995	4.5	2926	2.5	3.6	3	7.6	4.6
Samuel Dalembert	26	2001	5.0	1265	2.5	3.3	2.9	7.3	4.4

Doesn't look to be a common theme with these misses, and with the exception of Brook Lopez, it appears the models and scouts agreed on these players.