Tag: data

Making your app work with no data

Most applications, web, desktop or mobile, handle some kind of data. When we are developing them we generally generate some sample data to play with and we forget to ever run the application without it. The problem is that the first impression people will get of our app is without data. And first impressions are important.

In the application I’m building, Watu, we are resorting to just create some sample data for customers to play with. Making the application beautiful and meaningful without data is just too hard. It seems the guys at JetBrains spent some time thinking of this because RubyMine 4.0 shows this when there are no open files:

I think that simple change it’s a good step towards making the application nicer in the no-data scenario, making people happier, as well as making people more productive in it, making the application more useful.

I do wonder why they didn’t include the most useful of the shortcuts: ⌘⇧N. I think I press that more than any other. It pops up this little dialog:

in which you can type and it’ll search among all your files (and files inside the gems, that is, libraries, of your project if it doesn’t match anything in your project or if you enable by ticking the include non-project files):

What I really like, compared to other implementations of this feature, is that you can add more parts of the path, for example:

Without that feature, my productivity would drop a 10%. I’m not exaggerating, that’s my estimation, as I recently have to code using TextMate instead of RubyMine.

Before you send me the hate-mail, I know TextMate has a similar feature although I think not as advanced (not without plugins at least) but since the key shortcut was different, it was almost like it didn’t exist for me, so I experienced coding without that feature at all.

Another potentially useful way to find code is to use ⌘N which allows you to search for a class:

But since in a Rails projects most classes are in a file with the same name (but underscore instead of camel case) and the file dialog allows me to find views, wich the class dialog doesn’t, I never use the class dialog.

No… I’m not affiliated with JetBrains, makers of RubyMine in any way. I just love the tool and I wish more Ruby programmers would give it a try because I think they’ll also find it useful and the more people that are using it, the more resources JetBrains is likely to put into its development which ultimately benefits me. And they are cool guys, a pleasure to deal with every time I report a bug or ask for a feature.

Generating sample data

As I’ve said in previous posts, I like being able to generate sample data for the projects I’m working on very quickly. It allows new developers to get up to speed fast, and new developers to move faster.

When I don’t have a sample data generation method, I’m always scare to try whether, for example, deleting a project deletes all the todos in a project tracking system. Simply because I’ll have to generate that project and all todos by hand. Many times I end up not testing those destructive actions as often as I should.

The other reason while having a stable set of sample data is that you start to know it: “Hey! the users Paul and John are supposed to be on the same team, why I am not seeing them together? Something is broken”. To help with that I also use data that we already know. If I have teams with members I would create one team with John, Paul, George and Ringo called Beatles and another with Freddie, Brian, Roger and John called Queen. If you see Paul next to Freddie, something is broken.

To generate the sample data I use factories; which I also use to test instead of fixtures. If you are not familiar with factories, please, stop reading and go to check factory girl. I don’t care if you never come back to this blog if you start using factories instead of fixtures. Factories is so much better! But that’s probably repeated a thousand times over the web, so I’m not going to go into details.

In lib/tasks/data.rake I end up creating:

namespace :db do
  desc "Generate sample data for developing"
  task :sample_data => :environment do
    destroy_data()

    puts "==  Data: generating sample data ".ljust(79, "=")

    beatles = Factory.create :team, :name => "The Beatles"
    Factory.create :user, :name => "John Lennon", :team => beatles
    Factory.create :user, :name => "Paul McCartney", :team => beatles
    Factory.create :user, :name => "George Harrison", :team => beatles
    Factory.create :user, :name => "Ringo Starr", :team => beatles

    queen = Factory.create :team, :name => "Queen"
    Factory.create :user, :name => "Freddie Mercury", :team => queen
    Factory.create :user, :name => "Brian May", :team => queen
    Factory.create :user, :name => "John Deacon", :team => queen
    Factory.create :user, :name => "Roger Taylor", :team => queen

    puts "==  Data: generating sample data (done) ".ljust(79, "=") + "\n\n"
  end
end

For the implementation of destroy_data look at Deleting all records in a Rails project.

The problem with doing that with factories is that it is too silent. I like knowing what’s going on and for new developers it’s good to get a glimpse of the data. All users have the same password so after rake db:sample_data finishes, a new developer already know what email and password to use to log in. If you want to make it even easier, you can print out the password doing sample data generation.

The password is of course defined in the user factory:

Factory.define :user do |user|
  user.email { Factory.next :email }
  user.password "testing"
  user.password_confirmation "testing"
end

To be able to make factories verbose I created VFactory (for Verbose Factory of course) that you use just like Factory, but it prints out everything. This is its code:

# Verbose factory.
module VFactory
  def self.create *args
    human_factory_name = args.first.to_s.gsub("_", " ")
    if args.size > 1
      human_arguments = args.second.map { |name, value| "#{name}=>#{value.is_a?(Array) ? value.join(", ") : value}" }.to_sentence
      puts "-- creating #{human_factory_name} with #{human_arguments}."
    else
      puts "-- creating #{human_factory_name}."
    end
    Factory.create(*args).tap do |obj|
      puts "   -> done: #{obj}"
    end
  end
end

The output of this is more or less like this:

==  Data: generating sample data ==============================================
-- creating team with name=>The Beatles.
   -> done: #
-- creating user with name=>John Lennon and team=>#.
   -> done: #
-- creating user with name=>Paul McCartney and team=>#.
   -> done: #
-- creating user with name=>George Harrison and team=>#.
   -> done: #
-- creating user with name=>Ringo Starr and team=>#.
   -> done: #
-- creating team with name=>Queen.
   -> done: #
-- creating user with with name=>Freddie Mercury.
   -> done: #
-- creating user with with name=>Brian May.
   -> done: #
-- creating user with name=>John Deacon.
   -> done: #
-- creating user with name=>Roger Taylor
   -> done: #
==  Data: generating sample data (done) =======================================

If you are wondering why my objects look so pretty when printed, that’s because I always define a to_s for all models that contain the id and other important data. In this case it would be:

def to_s
   ""
end

That’s very useful for debugging. I also try to always have a name method in my models that give me something that represents the object and that I can show to the users.

The next step in data awesomeness would be, with one command, being able to download and import all production data. This really helps reproducing and debugging reported issues; specially when those issues are related to destructive changes.

Update: this is now a gem.

Deleting all records in a Rails project

During the initial phase of development of a Rails application I don’t use migrations as migrations but as table definitions. Until I deploy I feel free to modify the migration files as much as I want and I have one per table.

The downside of that is that the only way to apply the changes is to destroy all the tables and re-build them. I’ve explained how to do that in my post really resetting the database. The nice side effect of doing this is that you end up with a task that sets sample data to work with.

Being able to quickly set up sample data or download production data is very important. It helps new developers getting started with the project but it also allows you to play much more freely with the project, do destructive actions and then in a quick command have system reset to a known state. Once you have sample data you’ll probably become as addictive as I am to reseting.

But the truth is that 90% of the time you reset your data, you don’t need to nuke the database and re-create all records, you just need to delete all records and this is the code I use to do that:

def destroy_data
  puts "==  Data: Destroying all data ".ljust(79, "=")
  sql = ActiveRecord::Base.connection()

  sql.execute "SET autocommit=0"
  sql.begin_db_transaction

  if sql.adapter_name == "MySQL"
    sql.execute("/*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */")
    sql.execute("/*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */")
  end

  tables = sql.tables - ["schema_migrations"]

  tables.each do |table|
    puts "-- Deleting all for #{table}."
    # So far, disabling and enabling keys was not needed.
    #sql.execute("/*!40000 ALTER TABLE `#{table}` DISABLE KEYS */") if sql.adapter_name == "MySQL"
    record_count = sql.delete("DELETE FROM `#{table}`")
    #sql.execute("/*!40000 ALTER TABLE `#{table}` ENABLE KEYS */") if sql.adapter_name == "MySQL"
    puts "   -> done: #{record_count} reconds"
  end

  if sql.adapter_name == "MySQL"
    sql.execute("/*!40014 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS */")
    sql.execute("/*!40014 SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS */")
  end

  sql.commit_db_transaction

  puts "==  Data: Destroying all data (done) ".ljust(79, "=") + "\n\n"
end

Note that I’m not deleting anything in the table schema_migration. The output is more or less like this:

==  Data: Destroying all data =================================================
-- Deleting all for blogs.
   -> done: 4 reconds
-- Deleting all for posts.
   -> done: 10 reconds
-- Deleting all for users.
   -> done: 11 reconds
==  Data: Destroying all data (done) ==========================================

I also have some nice code to generate sample data, but that is for another post.