[Corpora-List] Coefficients in SVM models

Ken Litkowski ken at clres.com
Sat Aug 22 23:14:02 CEST 2015


Nice. I have noticed that very infrequent features do indeed get high weights in the models (as well as in other ML techniques such as decision trees or something like OneR), i.e., to quote the paper, "large weights correspond to small effects in the training data." Your way of looking at this provides a nuance that fits my intuition. Very helpful perspective.

Thanks,

Ken

On 8/22/2015 12:57 PM, Noah A Smith wrote:
> If your model is linear, you could try looking at features' "impacts":
>
> For each feature, calculate its average value in a validation set, and
> multiply by the weight (in the primal parameterization for your SVM).
> This tells you how much the model's linear score was affected, and in
> what direction, by this particular feature in this particular model,
> outside the training sample. (You could probably think of it as a
> frequentist estimate of "true" impact in the true population of
> instances.)
>
> I believe this idea is due to Brendan O'Connor, who wasn't a coauthor
> on the paper where we described it:
> http://homes.cs.washington.edu/~nasmith/papers/yano+smith+wilkerson.naacl12.pdf
> <http://homes.cs.washington.edu/%7Enasmith/papers/yano+smith+wilkerson.naacl12.pdf>
> . Note that you don't need to look at true labels to estimate impact,
> but you cannot fairly compare impact across models. More discussion in
> section 4.3 of the paper.
>
> (In that paper we were using logistic regression to learn the weights,
> but from the perspective of predictive model families, it is
> comparable to linear SVMs. I haven't thought very hard about backing
> out a similar kind of interpretation from an SVM with a non-linear
> kernel. With binary LR, the "linear score" is a log-odds. Things get
> trickier with multi-class models no matter which loss function you are
> optimizing during training.)
>
> Noah
>
>
> On Fri, Aug 21, 2015 at 1:58 AM, Behrang Q. Zadeh <me at atmykitchen.info
> <mailto:me at atmykitchen.info>> wrote:
>
> Hi Ken,
>
> Depending on the type of SVM kernel employed in your task, you can
> also exploit (alpha-stable) random projections to construct a
> vector space of lowered dimensionality (and somehow bypass the
> dimension reduction process such as explained in your email).
> Use a random projection-based method, e.g., random indexing in
> l2-regularised spaces, to construct a low-dimensional vector
> space, and then use this space for the training and classification
> (e.g., as suggested in [1]).
>
> Regards,
>
> Behrang
>
> [1] Sahlgren and Coster (2004). Using bag- of-concepts to improve
> the performance of support vector machines. URL:
> https://aclweb.org/anthology/C/C04/C04-1070.pdf
>
>
>
> *
> *
>
>
> On Fri, Aug 21, 2015 at 10:23 AM, Maximilian Haeussler
> <max at soe.ucsc.edu <mailto:max at soe.ucsc.edu>> wrote:
>
> Hi Ken,
> random thought: in an environment like Weka, R or sklearn, you
> can change your classifier to a regression or decision tree
> based classifier by changing just a single line in your code.
> The weights of the regression and the decision tree are easy
> to interpret.
> You could use the regression for the analysis of the feature
> influence, while still doing the final classification with the
> SVM, (in case that the SVM is really far superior to the
> regression).
> You could use a lasso or elasticnet regressor and increase
> alpha to remove features, just like the L1 parameter for SMV's
> suggested by Hady.
>
> cheers
> Max
>
> On Thu, Aug 20, 2015 at 8:36 PM, Ken Litkowski <ken at clres.com
> <mailto:ken at clres.com>> wrote:
>
> Are there any methods in computational linguistics for
> interpreting the coefficients in SVM models? I have 120
> nice preposition disambiguation models developed using the
> Tratz-Hovy parser, with an average of about 15,000
> features for each preposition. I'd like to identify the
> significant features (hopefully lexicographically
> salient). One such method (implemented in Weka) is to
> square the coefficients and to use this as the basis for
> ranking the features (the source of this method being a
> classic study by Guyon et al., 2002, in gene selection for
> cancer classification using support vector machines
> <http://link.springer.com/article/10.1023/A:1012487302797>).
> I'm extending these models (which make heavy use of WN)
> with other lexical resources, including FN, VN, and CPA.
> This will make the feature space even more
> hyperdimensional, so I'd like to pare them back in a
> principled way so I can see the potential contribution of
> these other resources.
>
> Thanks,
> Ken
>
> --
> Ken Litkowski TEL.:301-482-0237 <tel:301-482-0237>
> CL Research EMAIL:ken at clres.com <mailto:ken at clres.com>
> 9208 Gue Road Home Page:http://www.clres.com
> Damascus, MD 20872-1025 USA Blog:http://www.clres.com/blog
>
>
> _______________________________________________
> UNSUBSCRIBE from this page:
> http://mailman.uib.no/options/corpora
> Corpora mailing list
> Corpora at uib.no <mailto:Corpora at uib.no>
> http://mailman.uib.no/listinfo/corpora
>
>
>
> _______________________________________________
> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
> Corpora mailing list
> Corpora at uib.no <mailto:Corpora at uib.no>
> http://mailman.uib.no/listinfo/corpora
>
>
>
> _______________________________________________
> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
> Corpora mailing list
> Corpora at uib.no <mailto:Corpora at uib.no>
> http://mailman.uib.no/listinfo/corpora
>
>
>
>
> _______________________________________________
> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
> Corpora mailing list
> Corpora at uib.no
> http://mailman.uib.no/listinfo/corpora

-- Ken Litkowski TEL.: 301-482-0237 CL Research EMAIL: ken at clres.com 9208 Gue Road Home Page: http://www.clres.com Damascus, MD 20872-1025 USA Blog: http://www.clres.com/blog

-------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/html Size: 16499 bytes Desc: not available URL: <https://mailman.uib.no/public/corpora/attachments/20150822/3d0047e9/attachment.txt>



More information about the Corpora mailing list