That was an amazing post, Arcanaville! Yet I think I see a possible issue with the comparison of Tough and Weave on a Regen. I don't doubt the data! I just think we may be looking at the wrong data point for practical applications.
This may be difficult to explain without showing a simple graph, but I'll try.
Take the Y axis as a count of observations, and the X axis as health, where X can also be negative. I believe that in the tests you ran, the lowest that the scrapper's health goes in 30 minutes will follow a classic bell curve. The part of the bell curve that dips to 0 or below is the chance that the scrapper will die in that time frame.
To oversimplify what you said, when discussing Tough vs. Weave, you were looking at the amount of incoming damage where the scrapper still had a 99.9% chance of survival (in other words, where 99.9% of the bell curve was above 0.) While this isn't exactly what you stated, I think it is roughly equivalent to say that on a Regen, Tough significantly increases the % of bell curve above 0, while Weave has no effect on the % of bell curve above 0. So far so good.
Yet, since Weave must obviously block some incoming damage, it must also shift the center of the bell curve to the right. So how is it possible to shift the center of the bell curve to the right while not changing the % of the bell curve above 0? I would guess that for some reason, the bell curve has gotten wider with the addition of Weave. The net effect would be no increase in survivability. This seems perfectly consistent with your findings, as well as with the intuition that Weave should help in some way. It helps keep health higher on average, but doesn't help prevent death.
Tough comes out looking much better by comparison. I suspect that it both shifts the center of the bell curve to the right, while also narrowing the distribution. The net effect is a significant increase in survivability.
But these simulations are based on someone playing extremely safe. If I understood correctly, the analysis of Tough and Weave was done with 30 minute trials and a 99.9% survival rate. That's an average of one death every 500 hours of near constant fighting. I think most of us are willing to accept far more frequent deaths than that. But how frequent? One every 10 hours? Every five? Every hour? I'd suggest that most players are comfortable with something in that range.
For the sake of argument, let's say we can accept a death every hour of near constant fighting. If so, that's a 50% survival rate instead of 99.9%. I believe that will also put us in the center of the bell curve, not way off at one end. Half the bell curve is at 0 or below, half is above 0.
But as I already mentioned, I think Weave does shift the center of the bell curve to the right. It must, because it has a chance of making some attacks miss. Therefore, if we're accepting a death every hour of fighting, Weave is increasing our survivability. But by how much?
I suspect that by looking at the center of the bell curve, we're looking at the most simplistic version of the numbers. And from that standpoint, I think that Weave would be about the same as Tough. Weave is about a 6% defense to all, or about 12% damage mitigation. Tough is about 18% resist to smashing and lethal. If we assume that's about 2/3 of the damage in the game, then it's also about 12% damage mitigation. They behave quite differently, but I would still expect the center of the two bell curves to be at about the same point.
So I'd be curious to see what happens to Tough and Weave if you reran the simulations with a 50% survival rate. You did run some simulations later in the post with a 50% survival rate, but unless I missed it, none that compared Tough and Weave at those levels.
A death every hour may be more of a challenge than average play. So if I'm right about the shapes and positions of the bell curves, I would expect Weave to provide less benefit than Tough in average play, but a still noticeable benefit.
Of particular interest to me is what happens when pushing the performance envelope. What about PvP, the RWZ challenge, or a Storm Palace lap? In that case, I'd expect a much lower 30 minute survivability than 50%. I'm guessing that puts us at the opposite end of the bell curves. Perhaps in that case, Weave would help even more than Tough. It seems a bit far fetched to me, but then I wouldn't have expected Weave to be useless in boosting Regen survivability when playing extremely safe, and I was obviously wrong.
These are just my suspicions. But I'd definitely be curious to see what happens to the Tough vs. Weave comparison if you have the simulator look for the 50/50 point and beyond.
Thanks again for the great post.
(edit: I think I see a serious flaw in my reasoning. When I defined the curve, I stated that it was the lowest health reached in the half hour. Later, when arguing that the midpoint for Weave would be moved to the right, I discussed it as if it were the average health during the half hour. It seems perfectly reasonable that the midpoint of the lowest health during the half hour might hardly budge, regardless of the percent survival chosen, because a half hour is likely long enough to get a big unlucky streak with or without Weave. So it doesn't seem reasonable for me to argue that because it will prevent some damage, that the mid point of the curve will necessarily move to the right. I'd still be curious about the actual results, but I think my prediction was based on flawed logic.)
