Heroku run rails db:seed max file size

As we all know, seed files are required for some data-related applications. As I work with larger data sets, I'm encountering constraints that hosting providers have that my local machine does not. This article will explain the seed limitation when using git to push to Heroku, as well as how to overcome it.

The seed.rb standard is as follows:

heroku run rails db:seed        

This will usually work for users who only have a small seed file and are loading essential assets. We occasionally have large datasets and are developing a search application for users, so our seeds.rb file can grow quite large. The underlying issue here, which is not prompted by Heroku or Github, is that the maximum running seed file size is 50mb in memory. So you can run the command and get no errors, but the file is dropped on the backend, so our data isn't loaded into the database.

To solve this problem, I decided to separate the data into separate seeds.rb files. The main seeds.rb file can then call each seed file and run it, dividing the data into smaller sections.

Let's start by updating the code in our db/seeds.rb file.

# db/seeds.rb
Dir[File.join(Rails.root, "db", "seeds", "*.rb")].sort.each do |seed
  load seed
end        

Let's create a directory for the seed files next.

# From the main rails application
cd db
mkdir seeds        

We can now put all of our seed files in this folder. I followed the rails naming convention, so each file begins with seeds, followed by a number, and ends with the file extension.rb. I automated this process by running the following code against an array, but you can modify it to fit your needs.

records_split = [[],[],[],[],[],[],[],[],[],[]]
records_split.each_with_index do |data,index|
  outfile_json = File.join(__dir__, "seed#{index}.rb")
  File.write(outfile_json, data)
end        

This will generate output files with the appropriate naming convention.

Now, push your application to Heroku and run the seed file; the data should be loaded correctly.

Please let me know if I missed anything about Heroku documentation so that I can update this article. I'm sure there are more knowledgeable people out there, but this was my current solution.

THANKS!