Nice. I have noticed that very infrequent features do indeed get high
weights in the models (as well as in other ML techniques such as
decision trees or something like OneR), i.e., to quote the paper,
"large weights correspond to small effects in the training data." Your
way of looking at this provides a nuance that fits my intuition. Very
helpful perspective.
Thanks,
Ken
On 8/22/2015 12:57 PM, Noah A Smith wrote:
If your model is linear, you could try looking at features' "impacts":
For each feature, calculate its average value in a validation set, and
multiply by the weight (in the primal parameterization for your SVM).
This tells you how much the model's linear score was affected, and in
what direction, by this particular feature in this particular model,
outside the training sample. (You could probably think of it as a
frequentist estimate of "true" impact in the true population of
instances.)
I believe this idea is due to Brendan O'Connor, who wasn't a coauthor
on the paper where we described it:
[1]http://homes.cs.washington.edu/~nasmith/papers/yano+smith+wilkerson.
naacl12.pdf . Note that you don't need to look at true labels to
estimate impact, but you cannot fairly compare impact across models.
More discussion in section 4.3 of the paper.
(In that paper we were using logistic regression to learn the weights,
but from the perspective of predictive model families, it is comparable
to linear SVMs. I haven't thought very hard about backing out a
similar kind of interpretation from an SVM with a non-linear kernel.
With binary LR, the "linear score" is a log-odds. Things get trickier
with multi-class models no matter which loss function you are
optimizing during training.)
Noah
On Fri, Aug 21, 2015 at 1:58 AM, Behrang Q. Zadeh
<[2]me@atmykitchen.info> wrote:
Hi Ken,
Depending on the type of SVM kernel employed in your task, you can also
exploit (alpha-stable) random projections to construct a vector space
of lowered dimensionality (and somehow bypass the dimension reduction
process such as explained in your email).
Use a random projection-based method, e.g., random indexing in
l2-regularised spaces, to construct a low-dimensional vector space, and
then use this space for the training and classification (e.g., as
suggested in [1]).
Regards,
Behrang
[1] Sahlgren and Coster (2004). Using bag- of-concepts to improve the
performance of support vector machines. URL:
[3]https://aclweb.org/anthology/C/C04/C04-1070.pdf
On Fri, Aug 21, 2015 at 10:23 AM, Maximilian Haeussler
<[4]max@soe.ucsc.edu> wrote:
Hi Ken,
random thought: in an environment like Weka, R or sklearn, you can
change your classifier to a regression or decision tree based
classifier by changing just a single line in your code. The weights of
the regression and the decision tree are easy to interpret.
You could use the regression for the analysis of the feature influence,
while still doing the final classification with the SVM, (in case that
the SVM is really far superior to the regression).
You could use a lasso or elasticnet regressor and increase alpha to
remove features, just like the L1 parameter for SMV's suggested by
Hady.
cheers
Max
On Thu, Aug 20, 2015 at 8:36 PM, Ken Litkowski <[5]ken@clres.com>
wrote:
Are there any methods in computational linguistics for interpreting the
coefficients in SVM models? I have 120 nice preposition disambiguation
models developed using the Tratz-Hovy parser, with an average of about
15,000 features for each preposition. I'd like to identify the
significant features (hopefully lexicographically salient). One such
method (implemented in Weka) is to square the coefficients and to use
this as the basis for ranking the features (the source of this method
being a classic study by [6]Guyon et al., 2002, in gene selection for
cancer classification using support vector machines). I'm extending
these models (which make heavy use of WN) with other lexical resources,
including FN, VN, and CPA. This will make the feature space even more
hyperdimensional, so I'd like to pare them back in a principled way so
I can see the potential contribution of these other resources.
Thanks,
Ken
--
Ken Litkowski TEL.: [7]301-482-0237
CL Research EMAIL: [8]ken@clres.com
9208 Gue Road Home Page: [9]http://www.clres.com
Damascus, MD 20872-1025 USA Blog: [10]http://www.clres.com/blog
_______________________________________________
UNSUBSCRIBE from this page:
[11]http://mailman.uib.no/options/corpora
Corpora mailing list
[12]Corpora@uib.no
[13]http://mailman.uib.no/listinfo/corpora
_______________________________________________
UNSUBSCRIBE from this page:
[14]http://mailman.uib.no/options/corpora
Corpora mailing list
[15]Corpora@uib.no
[16]http://mailman.uib.no/listinfo/corpora
_______________________________________________
UNSUBSCRIBE from this page:
[17]http://mailman.uib.no/options/corpora
Corpora mailing list
[18]Corpora@uib.no
[19]http://mailman.uib.no/listinfo/corpora
_______________________________________________
UNSUBSCRIBE from this page: [20]http://mailman.uib.no/options/corpora
Corpora mailing list
[21]Corpora@uib.no
[22]http://mailman.uib.no/listinfo/corpora
--
Ken Litkowski TEL.: 301-482-0237
CL Research EMAIL: [23]ken@clres.com
9208 Gue Road Home Page: [24]http://www.clres.com
Damascus, MD 20872-1025 USA Blog: [25]http://www.clres.com/blog
References
Visible links
1. http://homes.cs.washington.edu/~nasmith/papers/yano+smith+wilkerson.naacl12.pdf
2. mailto:me@atmykitchen.info
3. https://aclweb.org/anthology/C/C04/C04-1070.pdf
4. mailto:max@soe.ucsc.edu
5. mailto:ken@clres.com
6. http://link.springer.com/article/10.1023/A:1012487302797
7. tel:301-482-0237
8. mailto:ken@clres.com
9. http://www.clres.com/
10. http://www.clres.com/blog
11. http://mailman.uib.no/options/corpora
12. mailto:Corpora@uib.no
13. http://mailman.uib.no/listinfo/corpora
14. http://mailman.uib.no/options/corpora
15. mailto:Corpora@uib.no
16. http://mailman.uib.no/listinfo/corpora
17. http://mailman.uib.no/options/corpora
18. mailto:Corpora@uib.no
19. http://mailman.uib.no/listinfo/corpora
20. http://mailman.uib.no/options/corpora
21. mailto:Corpora@uib.no
22. http://mailman.uib.no/listinfo/corpora
23. mailto:ken@clres.com
24. http://www.clres.com/
25. http://www.clres.com/blog
Hidden links:
26. http://homes.cs.washington.edu/%7Enasmith/papers/yano+smith+wilkerson.naacl12.pdf
27. mailto:me@atmykitchen.info
28. mailto:max@soe.ucsc.edu
29. mailto:ken@clres.com
30. http://mailman.uib.no/options/corpora
31. http://mailman.uib.no/options/corpora