ex4 - standard deviation of the error estimate

ex4 - standard deviation of the error estimate

guest (guest) 14 Dec 2013 17:17

Can you clarify what is r_t in the SD calculation described in the question?
What it t? The total number of tests (10% of the total yeast dataset size * 10 times) or just (10% of the total yeast dataset size)?

What values of C should we test? It takes a VERY long time to test large number of C's and we're not sure how to focus the test for important C's.

Re: ex4 - standard deviation of the error estimate

Elior_R 14 Dec 2013 19:16

Given a model we built using the training set, we would like to estimate the real error probability of the model. For that, we use the mean of the error rate over the test set. Of course, the estimation will probably be better (closer to the real error) if using more samples. We're using the SD of the estimation as a measurement for the expected estimation quality. Since the probabiliy for an error in the model is distributed Bernoulli with some p for error, the SD of the estimation is r_t, where t is the number of samples used for the estimation, i.e. the test set (the training set was used only for building the model). The SD of the error rate is given for each separate run of the k-fold, so we only need to decide what value to give as the SD of the error for the mean error rate across all the k executions - I just used the mean.

guest (guest) 14 Dec 2013 17:20

i also need the answers to these questions…

Can someone Please Help?

Thanks!!

guest (guest) 14 Dec 2013 17:28

I think they want the stdev of the accuracies within each 10fold crossval.
i.e. if you got 70%, 71%, … , 79% (for some d=k), then stdev = sqrt((70-74.5)^2 + (71-74.5)^2+…+(79-74.5)^2).

makes more sense this way.. and yeah, it'd be nice to get formal answers to both questions :)

marianos 15 Dec 2013 23:10

In the programming question of HW4 I made a mistake in the formula for the standard deviation of the estimated error.
The formula to use for estimating the the standard deviation of the estimated average A of N samples X_i to X_N (the respective N-fold cross-validation errors) is sqrt([1/(N-1))] *sum_i [(X_i-A)^2])
No points will be deducted for submissions that used the original (wrong) formula..

Last edited on 16 Dec 2013 08:42 by marianos Show more

guest (guest) 16 Dec 2013 07:08

Why 1/(N*(N-1))?
Dont you mean 1/N ?
(According to the definition of SD from wikipedia)

Just to clarify, you mean exactly what the guy above you said - sqrt((70-74.5)^2 + (71-74.5)^2+…+(79-74.5)^2), only we need to add 1/N [ or 1/(N(N-1)) ]?

marianos 16 Dec 2013 08:47

Fixed. N-1 in the denominator instead of N (Bessel's correction)