//
You are reading..
Datamining

Datamining Twitter Part 3 – Logging


I am a nervous person, so if my collection of tweets is running on the server I would somehow like to log what is going on so in case things go down I can at least know when it happened.

We will be using the logging library  .The logger gem is part of the standard package that ruby comes with so there is nothing to install. There is a nice comparison of loggers for ruby here (in German). 

Logging stuff in ruby is easy. You simply need this:

require 'rubygems'
require 'logger'

#since we want to write out to a file:
log = Logger.new("collect_tweets.log")

#You can use all of those different level errors to make your file more readable and see what is going on. 
log.debug("just a debug message") 
log.info("important information") 
log.warn("you better be prepared") 
log.error("now you are in trouble") 
log.fatal("this is the end...")

We will add those two callback methods to our client to log if errors are happening:


@client.on_delete do |status_id, user_id|
 log.error "Tweet deleted"
end

@client.on_limit do |skip_count|
 log.error "Limit exceeded"
end

And we will replace our output to console through the logger:


...
   rescue
    log.fatal "Could not insert tweet. Possibly db lock error"
    #puts "Couldnt insert tweet. Possibly db lock error"
   end
...

Now comes the trickest part. I would like the program to report to a log file every 10 minutes that it is up and running and doing fine.

loop collecting tweets
...
    time = Time.now.min
    if time % 10  == 0 && do_once
        log.info "Collection up and running"
        do_once = false
    elsif time  % 10 != 0
        do_once = true
    end
...

What will this do. Every time I insert a tweet I will check the time. Every 10 Minutes I want to once write my status. Notice that time.min % 10, would write the logging message to the log during that whole minute that it is runnung. So thats why we made a little flag do_once. It gets reset on between those 10 minutes. This should do just fine.

If we look in our log now we see:

I, [2010-05-25T09:10:02.436575 #2040]  INFO -- : Collection up and running
I, [2010-05-25T09:20:03.007758 #2040]  INFO -- : Collection up and running
I, [2010-05-25T09:30:03.002217 #2040]  INFO -- : Collection up and running
I, [2010-05-25T09:40:03.040313 #2040]  INFO -- : Collection up and running

Perfect. Now we can always look into this file and see how things have been. If the process somehow crashed we at least know when it happened.

In the next part I will show you how to use the deamons gem in combination with cronjob to make sure our process gets restarted if it somehow crashed.

Cheers Thomas

Advertisements

About plotti2k1

Thomas Plotkowiak is working at the MCM Institute in the Social Media and Mobile communication group which belongs to the University of St. Gallen. His PhD research in Social Media is researching how the structure of social networks like Facebook and Twitter influences the diffusion of information. His main focus of work is Twitter, since it allows public access (and has a nice API). Make sure to also have a look at his recent publications. Thomas majored 2008 in Computer Science and Economics at the University of Mannheim and was involved at the computer science institutes for software development and multimedia technoIogy: SWT and PI4. During his studies I focused on Artificial Intelligence, Multimedia Technology, Logistics and Business Informatics. In his diploma/master thesis he developed an adhoc p2p audio engine for 3D Games. Thomas was also a researcher for a year at the University of Waterloo in Canada and in the Macquarie University in Sydney. He was part of the CSIRO ICT researcher group. In his freetime thomas likes to swim in his houselake (drei weiher) and run and enjoy hiking in the Appenzell region. Otherwise you will find him coding ideas he recently had or enjoying a beer with colleagues in the MeetingPoint or Schwarzer Engel.

Discussion

No comments yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: