A Tidbit for Working with Rails and the Twitter API (Or, BigInt is Your Friend)
I’m working on a side project that taps the Twitter API and adds tweets into a database, and I ran into a funny problem. For some reason, there isn’t a lot written about this.
In Rails, the default database type is SQLite. It’s good for a lot of things, but I kept seeing people talk about how it wasn’t good at “concurrency” — having a lot of hits to the database at once. I don’t know how big of an issue that is with Rails, as Ruby is, I believe, limited in how much it can do at any given time. But I figured I’d switch it over to MySQL now, rather than down the road. And that’s where I hit my problem.
For some reason, my database wouldn’t save the tweet IDs as any number higher than 2147483647. I was trying to save tweets with IDs up around 4382700000. This is what it looked like in my script/console:
>> Scan.last.max_id = 4382700000
=> 4382700000
>> Scan.last.save
=> true
>> Scan.last
=> #<Scan id: 29, max_id: 2147483647, tweet_count: 2, created_at: "2009-09-25 19:59:08", updated_at: "2009-09-25 20:01:47">
Wha? Super weird.
Eventually, I figured out that this was the whole “Twitpocalypse” thing that people were talking about back in June. I didn’t care about it at the time, so I didn’t pay attention. But I put the pieces together.
Essentially, the problem is that the “integer” type in MySQL only goes up to the value “2147483647”. After that, you have to use a “BigInt” type. (SQLite doesn’t care about the number … it treats all integers the same. That’s why I wasn’t having problems on my development server, but it was choking on the production server — dev is running SQLite; production is now running MySQL.)
Anyway, changing it over in the database isn’t hard, although it took me a little bit to figure out the right syntax for BigInts. Turns out it’s “:bigint”. Der.
So here’s my migration file:
class ChangeColumnsToBigInt
def self.up
change_column :scans, :max_id, :bigint
change_column :tweets, :message_id, :bigint
end
def self.down
change_column :tweets, :message_id, :integer
change_column :scans, :max_id, :integer
end
end
And now everything works like cake.
Oh. This post was the most helpful post I found on this, although I didn’t actually find it until the very end of my research and futzing.
