Rails 大数据处理

so_brave

Rails 大数据处理

If you want to do a large data query such as finding all the 10,000,000 users to send email to them, you should use batched finder to avoid eating too much memory.

Imagine you have a newsletter system which is very famous and has 10,000,000 users. Every Monday morning, the system will send emails to all of the users.

In General
You may find all the users, then send emails to them one by one

User.all.each do |user|
  NewsLetter.weekly_deliver(user)
end
It may work, but do you know there are 10,000,000 users? This code snippet will find all of the users and create a ruby object for each user row in table. The server is unhappy due to too much memory is eaten by your newsletter application.

Improvement
From rails 2.3, find_each and find_in_batches are available for batched finder. We can use them to improve the performance of the newsletter application.

User.find_each do |user|
  NewsLetter.weekly_deliver(user)
end
Using find_each, the application only finds 1,000 users once, yield them, then handle the next 1,000 users, until the last 1,000 users. That means the application will only load 1,000 user objects into memory each time, the server is happy now.

1,000 is the default batch size, if you think it is small/big to you, you can use the :batch_size option to change it.

find_in_batches is similar to find_each except that it yields the array of objects

User.find_in_batches(:batch_size => 5000) do |users|
  users.each { |user| NewsLetter.weekly_deliver(user) }
end
This method is very useful for large data query, saving your memory and time.



摘自：http://rails-bestpractices.com/p ... or-large-data-query

Asp菜菜

嗯嗯

2gua

不错。