[Corpora-List] Building a corpus from Twitter & Tw's privacy concerns

Leon Derczynski leon at dcs.shef.ac.uk
Tue Jul 16 18:49:06 CEST 2013


Dear John,

You are right, it is a very awkward distribution format. Corpora of some size become very difficult to gather after a point. However, tweet decay is not so much of a problem for many use-cases (of course when tweets are deleted that are the only example of a particular phenomenon in a corpus, the deletion is of greater impact). It seems that tweets decay at a reasonably stable rate after the first hour's spam filtering (or whatever) has been done.

Based on observations over the Trendminer garden hose feed, which collects us about 1.2TB of tweets each day, and the findings of Petrovic, Osborne and Lavrenko (2013) "I Wish I Didn’t Say That! Analyzing and Predicting Deleted Messages in Twitter", you can in fact work out a reasonably accurate model for predicting tweet "decay" through deletion in a reasonably-sized corpus:

http://derczynski.com/leon/tweet-decay.html

This supports the idea that a corpus a few months old and a few years old will look roughly the same - it's quite rare that one would go back and delete a message after a few years. So the tweet data becomes "stabler" with time. Which is good, because it may take months to retrieve via the API ;)

All the best,

Leon

On 16 July 2013 17:44, John D. Burger <john at mitre.org> wrote:


> There appears to be no legal reason you can't collect a corpus of tweets.
> However, per Twitter's Terms of Use you cannot redistribute the tweets to
> others. A common practice is to instead distribute the tweet IDs, which
> other people can use to fetch the tweets using Twitter's API. This is how
> NIST "distributes" the data in their Tweets2011 corpus:
>
> http://trec.nist.gov/data/tweets/
>
> This is less than optimal for research, though, since in the interim some
> of the Twitter users may have deleted tweets in the collection. For a
> sufficiently large corpus, this means that anybody else attempting to use
> the same data at a later date will almost certainly end up with a subset of
> your corpus. As far as I know, however, this is currently the only legal
> method for sharing tweets.
>
> - John Burger
> MITRE
>
> On Jul 16, 2013, at 10:51 , M.E.Sciubba wrote:
>
> Dear ListMembers,
>
> I'd like to create a corpus of Italian twits, but searching online I found
> out that it is not possible anymore because Twitter has changed its privacy
> settings.
>
> Has any of you tried to build a Twitter corpus and how?
>
> Any suggestion will be much appreciated (considering that I am not a
> programmer, though).
>
> Best,
>
> Eleonora
>
>
> ------------------------------
>
> Dr. Maria Eleonora Sciubba
> Associate Researcher
> Archivio di LInguA Spontanea
> tel. +32 16 3 24795
> cell +32 483 616 114
>
> KU Leuven – Faculty of Arts****
>
> Department of French, Italian and Comparative Linguistics****
>
> Blijde-Inkomststraat 21, PO BOX 3308****
>
> B - 3000 Leuven
>
> http://www.kuleuven.be/wieiswie/nl/person/00088846
>
>
>
> ------------------------------
> *Be green. Keep it on the screen*
> _______________________________________________
> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
> Corpora mailing list
> Corpora at uib.no
> http://mailman.uib.no/listinfo/corpora
>
>
>
> _______________________________________________
> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
> Corpora mailing list
> Corpora at uib.no
> http://mailman.uib.no/listinfo/corpora
>
>

-- Leon R A Derczynski Research Associate, NLP Group

Department of Computer Science University of Sheffield Regent Court, 211 Portobello Sheffield S1 4DP, UK

+45 5157 4948 http://www.dcs.shef.ac.uk/~leon/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/html Size: 6804 bytes Desc: not available URL: <https://mailman.uib.no/public/corpora/attachments/20130716/546fa2ac/attachment.txt>



More information about the Corpora mailing list