[Corpora-List] Coefficients in SVM models

Noah A Smith nasmith at cs.cmu.edu
Sat Aug 22 18:57:09 CEST 2015


If your model is linear, you could try looking at features' "impacts":

For each feature, calculate its average value in a validation set, and multiply by the weight (in the primal parameterization for your SVM). This tells you how much the model's linear score was affected, and in what direction, by this particular feature in this particular model, outside the training sample. (You could probably think of it as a frequentist estimate of "true" impact in the true population of instances.)

I believe this idea is due to Brendan O'Connor, who wasn't a coauthor on the paper where we described it: http://homes.cs.washington.edu/~nasmith/papers/yano+smith+wilkerson.naacl12.pdf . Note that you don't need to look at true labels to estimate impact, but you cannot fairly compare impact across models. More discussion in section 4.3 of the paper.

(In that paper we were using logistic regression to learn the weights, but from the perspective of predictive model families, it is comparable to linear SVMs. I haven't thought very hard about backing out a similar kind of interpretation from an SVM with a non-linear kernel. With binary LR, the "linear score" is a log-odds. Things get trickier with multi-class models no matter which loss function you are optimizing during training.)

Noah

On Fri, Aug 21, 2015 at 1:58 AM, Behrang Q. Zadeh <me at atmykitchen.info> wrote:


> Hi Ken,
>
> Depending on the type of SVM kernel employed in your task, you can also
> exploit (alpha-stable) random projections to construct a vector space of
> lowered dimensionality (and somehow bypass the dimension reduction process
> such as explained in your email).
> Use a random projection-based method, e.g., random indexing in
> l2-regularised spaces, to construct a low-dimensional vector space, and
> then use this space for the training and classification (e.g., as suggested
> in [1]).
>
> Regards,
>
> Behrang
>
> [1] Sahlgren and Coster (2004). Using bag- of-concepts to improve the
> performance of support vector machines. URL:
> https://aclweb.org/anthology/C/C04/C04-1070.pdf
>
>
>
>
>
> On Fri, Aug 21, 2015 at 10:23 AM, Maximilian Haeussler <max at soe.ucsc.edu>
> wrote:
>
>> Hi Ken,
>> random thought: in an environment like Weka, R or sklearn, you can change
>> your classifier to a regression or decision tree based classifier by
>> changing just a single line in your code. The weights of the regression and
>> the decision tree are easy to interpret.
>> You could use the regression for the analysis of the feature influence,
>> while still doing the final classification with the SVM, (in case that the
>> SVM is really far superior to the regression).
>> You could use a lasso or elasticnet regressor and increase alpha to
>> remove features, just like the L1 parameter for SMV's suggested by Hady.
>>
>> cheers
>> Max
>>
>> On Thu, Aug 20, 2015 at 8:36 PM, Ken Litkowski <ken at clres.com> wrote:
>>
>>> Are there any methods in computational linguistics for interpreting the
>>> coefficients in SVM models? I have 120 nice preposition disambiguation
>>> models developed using the Tratz-Hovy parser, with an average of about
>>> 15,000 features for each preposition. I'd like to identify the significant
>>> features (hopefully lexicographically salient). One such method
>>> (implemented in Weka) is to square the coefficients and to use this as the
>>> basis for ranking the features (the source of this method being a classic
>>> study by Guyon et al., 2002, in gene selection for cancer
>>> classification using support vector machines
>>> <http://link.springer.com/article/10.1023/A:1012487302797>). I'm
>>> extending these models (which make heavy use of WN) with other lexical
>>> resources, including FN, VN, and CPA. This will make the feature space even
>>> more hyperdimensional, so I'd like to pare them back in a principled way so
>>> I can see the potential contribution of these other resources.
>>>
>>> Thanks,
>>> Ken
>>>
>>> --
>>> Ken Litkowski TEL.: 301-482-0237
>>> CL Research EMAIL: ken at clres.com
>>> 9208 Gue Road Home Page: http://www.clres.com
>>> Damascus, MD 20872-1025 USA Blog: http://www.clres.com/blog
>>>
>>>
>>> _______________________________________________
>>> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
>>> Corpora mailing list
>>> Corpora at uib.no
>>> http://mailman.uib.no/listinfo/corpora
>>>
>>>
>>
>> _______________________________________________
>> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
>> Corpora mailing list
>> Corpora at uib.no
>> http://mailman.uib.no/listinfo/corpora
>>
>>
>
> _______________________________________________
> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
> Corpora mailing list
> Corpora at uib.no
> http://mailman.uib.no/listinfo/corpora
>
>
-------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/html Size: 7531 bytes Desc: not available URL: <https://mailman.uib.no/public/corpora/attachments/20150822/2dbd35d5/attachment.txt>



More information about the Corpora mailing list