Humble Technologies Freelance Web Development - blog

Backup Your Rails App To Amazon S3

2008-05-30T15:31:00Z

Backup is one of those boring subjects that is really important. Lots of hosts claim to backup your data but where is it? You basically have to trust that they are doing this and trust that you can get hold of it when you ask them. In my book thats a lot of trust with your oh so valuable data so best to do your own backups.

Amazon's S3 service is a great options for data backups and thanks to the brilliant AWS::S3 gem its not difficult to implement either. I thought I'd post some of my work here as it is a nice simple example of using the AWS:S3 gem for backups using rake. After installing the gem with sudo gem i aws-s3 I ran script/console and manually created a couple of S3 buckets for my data (these are kind of like folder). The nice thing about doing this manually is you can check all your S3 credentials are working OK as well as the gem is installed correctly.


AWS::S3::Base.establish_connection!(
      :access_key_id     => 'your key id',
      :secret_access_key => 'your secret key'
  )

AWS::S3::Bucket.create('humble_daily_backup')
AWS::S3::Bucket.create('humble_weekly_backup')

Great now we have our buckets. If something went wrong here you probably need to check your keys and that your bucket names are unique.

Now create a rake file in your rails lib/tasks folder, something like utils.rake and then paste the following code into it. The idea is that it has daily rolling backups so the last 7 days of backups are kept and also persistent weekly backups. For the daily backups I use the week day name as the file name that way I can easily delete the file from 8 days ago and create the new one. For weekly backups we just use the date.


namespace :utils do
  
  desc "Backup the databases to Amazon S3 daily"
  task(:daily_backup) do
    backup_to_s3('humble_daily_backup', Date.today.strftime("%A"))
  end
  
  desc "Backup the databases to Amazon S3 Weekly"
  task(:weekly_backup) do
    backup_to_s3('humble_weekly_backup', Date.today.strftime("%Y-%m-%d"))
  end
    
  private
  
  def backup_to_s3(bucket, backup_name)
    
    require 'yaml'
    require 'rubygems'
    require 'aws/s3'

    app_dir = '/full/path/to/your/app/current'
   
    dump       = "humble_#{backup_name}.sql.gz" 
    config = YAML::load(open(app_dir + '/config/database.yml'))['production']
    cmd    = "mysqldump -u #{config['username']} -p#{config['password']} -h localhost --add-drop-table --add-locks --extended-insert --lock-tables #{config['database']} | gzip -cf9 > #{dump}" 

    puts 'Getting ready to create a backup'
    `#{cmd}`
    
    puts 'Backup created, starting the transfer offsite'
    AWS::S3::Base.establish_connection!(
      :access_key_id     => 'your key id',
      :secret_access_key => 'your secret key'
    )
    
    AWS::S3::S3Object.delete(dump, bucket) if AWS::S3::S3Object.exists?(dump, bucket) 
    AWS::S3::S3Object.store(dump, open(dump), bucket)
    
    puts 'Offsite transfer completed'
    
    cmd       = "rm -f #{dump}" 
    `#{cmd}`
    puts 'Local copy of database dump deleted'
  
  end

So now you have this rake task you can run them manually with


rake utils:daily_backup
rake utils:weekly_backup

To make them run automatically which is obviously what you want for daily and weekly backups we need to add some cron jobs. So here they are...


0 0 * * * bash -c 'cd /full/path/to/your/app/current/; /usr/local/bin/rake utils:daily_backup >> /full/path/to/your/app/current/log/daily_backup.log 2>&1'
0 0 * * 0 bash -c 'cd /full/path/to/your/app/current/; /usr/local/bin/rake utils:weekly_backup >> /full/path/to/your/app/current/log/weekly_backup.log 2>&1'

Now the icing on the cake. If your on a mac try this great little S3 GUI. We can now check that our backups are working properly with this GUI and grab data backups if and when we need to recreate our database.

Rails Free Text Search: Sphinx or Ferret

2008-05-02T16:24:00Z

I’ve now had chance to give both Ferret and Sphinx a good flogging on recent Rails projects and think its time to put them head to head.

Ferret

Ferret along with Acts As Ferret offers a developer friendly Rails way of adding free text search. It closely follows the Lucene syntax so if you want to go beyond the normal keyword search and add filters, data rangers, fuzzy strength, flags its all possible with a bit of swatting up on the Lucene here. The reason it is so developer friendly is that it uses ActiveRecord callbacks to update the indexes. That gives 2 key advantages:

Indexes are always up to date if all your relevent database changes are happening through your rails app.
The indexer will call ActiveRecord attributes/methods rather than looking directly in the database which means where you have complex associations you are trying index in someway you can just convert the relevant data into a comma separated string or something like that.

So ferret is pretty straight forward to use thanks to a great plugin, a great tutorial and a background server which is vital in a production environment. But and it seems to be a big BUT does it work in busy production environments. Ezra from Engine Yard who knows more about these things than most says they have had constant trouble with ferret on deployed applications and have helped their clients move to other solutions. I on the other hand have had no problems in production environments and found it works fine as long as capistrano correctly stops and starts the server between code updates to avoid index corruption. Also looking around the web plenty of other people seem to have success with ferret too. Perhaps its just high traffic sites where it becomes a problem or perhaps recent ferret releases have fixed the issues but I’m still not convinced that this solution is dead.

Sphinx

Sphinx is the new kid on the block in the Rails world and already has 3 well used plugins. I hope they converge eventually as it seems wrong 3 people/teams working separately on the same goal. Anyway for reference the plugins are ThinkingSphinx, Sphincter and finally my chosen companion UltraSphinx.

I found this a lot harder to setup probably as the tutorials out there are not as clear as the Rails Envy ferret tutorial (its on my to-do list) and perhaps because its relationship with Rails even with a well developed plugin seems less natural than ferrets. Sphinx is powerful, fast and robust several things that ferret supposedly isn’t. But I have a couple of issues with it.

The code required to get associations indexed requires some lengthy sql joinery.
A bigger issue is the fact that you need to reindex with a cron job periodically. This means for the period of time in between reindexes the search results may be out of date. Now for many sites this isn’t an issue especially traditional publishing style sites. But for web application it can be. If you add data you expect to be able to find it again seconds later. Thanks to delta indexing we can run frequent delta indexes to keep indexes more up to date without having to reindex the whole database but still there are likely to be a couple of minutes of lag and thats not always acceptable depending on the application.

Conclusion

So recently in the Rails world I have read a lot of Sphinx good Ferret bad posts and now I’m not sure its that simple. I think if you need a fast and robust solution for high traffic sites then sphinx may be best but if you don’t have huge traffic or you have web apps where instant indexing is a real useability need then I still think ferret has legs.

Quickies: Rails & RSS

2008-01-11T10:36:00Z

When I was adding RSS to a site recently I knew what I needed to do was pretty straight forward but I ended up having to consult a fair few sites to get the lowdown on RSS and Rails. I’ve tried to make it easier for everyone else by summarizing RSS and Rails here.

URLs and respond_to

So first of all we need the rss feed url. If it contains content relating to a HTML page then the Rails way is to put the RSS stuff into the same page action and request the URL with the’.rss’ extension. This means we can use a respond_to block in the action and the feed url will be detected as an rss format request. So in your action you need something like…


respond_to do |format|
  format.html
  format.rss { render :action => "feed.rxml", :layout => false  }
end

Build the RSS

I use the RSS2 specification it seems to be widely adopted up to date and does what I need. The best explanation of this that I found is here . New create the feed.rxml template in the correct folder. Here is an example of the rxml template.


xml.instruct! :xml, :version=>"1.0", :encoding=>"UTF-8" 
xml.rss(:version=>"2.0"){
  xml.channel{
    xml.title(@title)
    xml.link(url_for(:only_path => false, :controller => 'items'))
    xml.description(@description)
    xml.language("en")
    for item in @items
      xml.item do
        xml.title(item.title)
        xml.link(url_for(:only_path => false,
          :controller=>'items', :action=>'show', :id=>item))
        xml.description(item.summary)
        xml.pubDate(item.created_at.rfc2822)
        xml.guid(url_for(:only_path => false,
          :controller=>'items', :action=>'show', :id=>item))
      end 
    end  
  }
}

A couple of things to note here. First of all the tidy xml builder that rxml uses by default. It makes it super simple to build xml documents. Secondly how simple RSS2 really is its basically just a bunch of titles, links and descriptions, very generic. Thirdly notice the UTF-8 directive at the top this may not be necessary, it basically depends on the way your data is stored in the database but if you have UTF-8 data make sure this is set correctly or you’ll have all sorts of problems with strange characters.

Associating feed with a web page

If we are on the html page which has an associated feed we really want the browser to detect that there is a feed. To do this you need to add a link to your html head element.


<head>
   ...
  <link href="<%= url_for :only_path => false, 
                      :controller=>'items', :format=>'rss' %>" 
   rel="alternate" type="application/rss+xml" />
   ...
</head>

This is likely to be in your layout so you may be better using content for. In your rss associated template…


<% content_for(:rss) { url_for :only_path => false, 
       :controller=>'items', :format=>'rss' } %>

and then in the layout something like…


<head>
   ...
  <link href="<%=  yield(:rss)  %>" 
     rel="alternate" type="application/rss+xml" />
   ...
</head>

This way we keep the action specific directives in the action template but still have the rss link in the correct place which is defined in the layout. Think thats about it. Hope it helps someone.

Firefox Add-ons for Web Development

2008-01-09T16:25:00Z

So if you’re a web developer you’re using Firefox right? Why? Well it offers great standards compliance and a ton of useful add-ons for web development. Here are the add-ons that adorn my browser and make my life as a web developer a whole lot easier.

Firebug – This is the most important one. When it comes to markup and CSS I pretty much live in Firebug. Particularly great for debugging CSS and Javascript problems. It includes a Javascript console, context sensitive HTML and CSS source inspection and super useful request logging for debugging those pesky AJAX problems. It does a whole lot more but they’re the tools I find most useful.

Web Developer Tools – This offers a huge number of functions. It’s difficult to know where to start. Basically I find it super useful for switching Javascript and CSS on and off, outlining and validating.

MeasureIt – This is pretty basic but useful. It measures stuff. So if you’re on a web page and wondering how wide the layout is, how wide the column is, how big an image is then MeasureIt tells you. Give it a try.

ColorZilla – Ever wondered the exact hex for a color on a web page and sick of screen grabbing it and using the color picker in Photoshop? If so this is for you. It will tell you the color of any part of a web page shown in the your browser which is really useful when you are coming up with color palettes for new designs.

Selenium IDE – This is a super useful acceptance testing tool. Basically you get it to record your actions as you go through your website or web application, click links, populate forms, log in etc. The recordings are saved as scripts that you can then easily tweak and replay. I use this heavily before deploying new releases to make sure everything is looking and functioning as I expect. There is a lot to say about Selenium and using it with Rails projects so look out for more on this in a future post. But with or without Rails its a great website test tool.

Total Validator – We are all trying to keep are markup W3C compliant these days (well most of us) and thats a very good thing. But I found it hard to find a decent tool for validating local files in my development environment. Total Validator has a firefox extension that can bundle up the source and post it to its validation service where it not only validates your markup but also gives some useful tips for correcting any errors. This has been a life saver for me.

IE Tab – Needs no explanation really. Installs a button in you status bar that switches the rendering window between Internet Explorer and Firefox. Dead useful for ironing out those browser compatibility issues.

Alexa Sparky – OK so this isn’t a developer add-on as such but its really interesting if you are curious about the popularity trends of sites you visit and that kind of information comes in useful if your about to embark on your own project.

Freelancing On Rails Part 1

2007-07-30T16:18:00Z

I really can’t imagine freelancing with any other technology than Rails right now. Why?

I need to be able to make quick enhancements and show clients with in a matter of hours on a staging server. Ruby and Rails are concise, Capistrano is ultra convenient and hey presto.
I’m generally a 1 man team and I need code that one man can manage. To me that means it needs to be clean, concise and expressive.
I need to reuse. I wouldn’t be able to offer my clients competitive estimates if there wasn’t a ton of code out there that I knew was stable, clean and relevant to my projects. Thanks to the great Rails community and the Rails plug-in architecture there is.
These days its a requirement for me that I enjoy what I do and have a passion and feeling of achievement from it. Optimized for programmer happiness Rails gives me this.
I like to concentrate on design and useability. Although programming is my original trade my gift and passion is for creating useful applications. This basically means talking to end users, empathising with end users and listening, no I mean really listening to my clients. Such a concise high-level language like Ruby coupled with the pragmatic nature of rails means I can concentrate on functionality, useability, beauty, aesthetics and the rest of it rather than code, code, and more code.
Predictability. Well I’m not sure software development will ever be predictable, requirements evolve, business change, technologies change so predicting exact time frames and prices for projects is not easy, no not even with Rails. But Rails enables me to be agile, flexible and pragmatic so requirement changes don’t lead to a project being completely knocked off course. Changes will happen, clients are never completely happy with the first thing you show them that’s just the way it is. So I develop as part of the communication process get things out there and iterate, iterate and iterate and the code stays clean.
I am a designer, CSS and DHTML guy too and there are only so many hours in the day. Using a speedy framework like Rails means I really can share my time between wearing these various hats without drowning in complexity.

Ahh. Glad I got that off my chest.

Caching the night away

2007-07-30T12:06:00Z

One of my rails apps. sizeasy has recently been getting consistently more traffic thanks to a bit of search engine optimisation. I noticed a performance hit on my server with the more constant use and decided it was high time I got serious about caching. I could have done this much earlier but I’m a strong believer in the scale later philosophy , to me as long as you are aware of ways you can scale and optimise and you don’t tie yourself up to start with then you just get more bang for your buck if you do it when it needs doing. Anyway turns out caching in rails is another one of those “is that all I have to do” situations even with my slightly uncommon requirements…

Action Caching

First off sizeasy does lots of server side dynamic image generation and it won’t surprise you to hear that this is slightly memory and cpu intensive and something you want to minimize to the extreme. So basically if I’ve created an image before I just want to cache it and never have to create it again. Seems sensible enough but although Ive used rails action caching before I didn’t think it would work for image outputs. How wrong I was simply do

caches_action :draw_cube

and it works out of the box. Well almost. What makes a cube image unique is its dimensions and its colour which are all passed over as parameters on the image request. By default the action caching in rails doesn’t take into account parameters on the query string so I needed something that did. Plugins, plugins, plugins, rather like Firefox and its extensions something that makes rails so attractive is its community and in turns its plug-ins. So often I find someone has already done it before and I can just plug it in. Were still talking quite low level here not application level e.g. plug-in a blog and plug-in a forum (however a framework for making this kind of thing possible would be awesome) no more coder level chunks that can still save serious developer resource. So anyway back to caching, yes the plug-in I used was the query string action caching plug-in which once pluged in requires no code changes at all it just means now the action_cache mechanism uses the entire url including query string to create and uniquely identify cached content.

Page Caching

So that was the main hurdle crossed query string specific image caching. My app started flying. But I could do more. There are several pretty much static pages in the app for help, instruction, buzz etc and these were still hitting the application server. Well these situations are even easier to deal with. Just a simple one liner and after the first time these guys are just served up as static pages.

caches_page :buzz, :tips, :featured

Partial Caching

People hit the homepage a lot and often its there first impression of the site. We want this to load quickly and make an impression rather than chug away loading and giving them plenty of time to hit the back button and try somewhere else. On sizeasy’s homepage we have a latest and most popular comparisons list which hit the database to populate these lists. Again it seems a waste to hit the database every time a homepage is displayed when these views don’t change anywhere near as often as the page gets displayed. Enter, fragment caching. Simply add a cache block around an arbitrary piece of template code and this chunk gets cached and not regenerated until the fragment is explicitly expired. So the code looks something like this…

<% cache(:action => 'sizeup', :part=>"latest" ) do %>

<%#Put your fragment cacheable code in here  %>

<% end %>

Now obviously we need this to change whenever new comparisons are saved so in the action that saves a new comparison we simply need to add a line of code to expire this cache fragment.

expire_fragment(:action=>'sizeup', :part=>'latest')

Now that wasn’t so hard was it.

Summary

So in Rails we have page caching, action caching and fragment caching and by default these caches are all stored as files. Page caching is the quickest as requests don’t hit the application server at all, they are simply served up as static files. Then there is action caching which does hit the application server but if you have database lookups or image generation then you are likely to see massive speed increases from action caching rather than not caching. Fragment caching is also very useful especially in many web 2.0 sites that like to have panels with lists of recent, popular, random stuff on various pages. Fragment caching can easily minimize the amount of times your server has to do database lookups and partial page generation. All in all caching is dead simple in Rails but still I think optimise later is a great approach to getting things done. So caching for many of us comes down the line. Just one more thing.

If you love Capistrano like I do then you may want to maintain your cache between incremental releases. So I added a rake task for creating symlinks and called it in deploy.rb :after_before_update. Here is the rake task…

desc "symlink to shared" 
task :symlink_to_shared do
  rel_path = ENV['RELEASE_PATH']
  share = 'shared'
  dir = ENV['DIR']
  share_path = File.expand_path("#{rel_path}/../../#{share}/#{dir}")
  unless File.exists?(share_path)
    system "mkdir #{share_path}" 
  end
  puts "ln -s #{share_path} #{rel_path}/#{dir}" 
  system "ln -s #{share_path} #{rel_path}/#{dir}" 
end

And here is where I call it

task :after_update_code, :roles => :app do
  run <<-cmd cd>rake symlink_to_shared DIR=cache RELEASE_PATH=#{release_path} &&
    rake deploy_edge REVISION=#{rails_version}
  CMD
end

Rails resources: Screencasts

2007-06-07T16:11:00Z

One thing I love about rails is the variety of great resources in a variety of useful mediums. There’s podcasts, books, a ton of great blogs, wiki’s, forums, chat rooms, mailing lists, user groups and more. Sometimes after a hard days coding I’m a bit too jaded to carry on hacking away or reading technical books but one thing I find I can do with five minutes to spare is listen to podcasts and screencasts. The great thing about these is you can watch them on your ipod anyway anytime. There are 3 great screencast resources as far I’m aware.

Introduction to Rails: If your new to Ruby on Rails and want to see what its all about then there is no better place to start than here with the now infamous how to write a blog application in 15 minutes. Its a great way to get a feel for Rails but it sure is neat marketing and by no means do you have a production ready fully functional blog in 15 minutes so take it with a pinch of salt.

Railscasts: If you want some great, professional bite size Rails lessons then one of my favourite resources right now is Ryan Bates’ Railscasts . These are just awesome, short tips and tricks for rails. But what I like most about them is how relevant they are. I find you can read books and blogs with great technical insight and then get started on projects and not know how to do seemingly trivial things. Ryans screencasts offer great, tidy answers to real world implementation problems. One thing I learned recently was the fantastic rails addition to Array things.collect(&:property) syntax as short hand for collecting the result of a call to the property method on all the “things”. Rails doesn’t stop surprising me. These look great on your iPod video too.

PeepCode: Last but best of all are the peepcode screencasts by Rails socialite Geoffrey Grosenbach. These are in depth tutorials which explain core Rails concepts from novice to expert in 1- 1.5hr screencasts. They include so far Capistrano, PrototypeJS, TDD and RESTful rails among other things. These are detailed and thorough and he uses experts in the given areas to advise him so he gets it all spot on and bang up to date. These are really valuable each one probably equivelent to a half day training session for which you’d pay a fair amount of cash. But these babies are only $9 a pop which is an absolute bargain for what they contain. Again they come in iPod video format or .mov files. Because they are quite involved you may find following along with the code on an iPod a little tricky as it is barely legible. Still fantastic resource and worth every penny to speed up the ruby and rails learning curve.

Well that’s about it for screencasts. As a community we are so lucky to have people that provide this quality of resource, cheers guys! Take a look at them, subscribe and enjoy.

Rails resources: Books

2007-04-24T21:52:00Z

I'm going to write a mini series of blog posts about some of my favourite Rails resources. There more and more great stuff appearing everyday and I just want to help people choose the right resources so they get the most knowledge with the least headaches. Well to kick things off I'm going to tell you about my favourite Ruby and Rails paper based companions that have helped me get up to speed throughout the last year or so.

Agile web development with Rails - (4/5) This is the original guide to Rails written by Dave Thomas of pragmatic programmer fame and DHH himself. I would say make sure you get the 2nd edition. I got the first back in early 2006 long before the second was released but the second book is a must to get you up to date with all the latest 1.2 features including the much loved RESTfulness. Its a got a great getting started feel to it but also acts a pretty good reference when it digs into more details further on in the book.

Programming Ruby (The Pickaxe) - (5/5) Another Dave Thomas book is generally regarded as the definitive Ruby manual and when you try and lift the thing let alone read it you can see why. It starts off with a pretty good introduction to Ruby but it moves along quickly and is definitely for software developers with a bit of general OO experience under the belts. I have found it indispensible as a reference book. Ruby has a lot of power and a lot of libraries and yes you can get 10 lines of Java into 1 line of Ruby but theres a good chance you may have to look something up in here first to do it!

Rails Recipes - (5/5) This is a great book. Its not one for beginners but as soon as you have been involved in some real world Rails projects you will appreciate a lot of the incredibly tidy solutions to common problems that this book offers. My only issue is trying to remember all the great stuff in the book so I can use it rather than reinventing the wheel in a less optimal way myself. Looking forward to the next Chad Fowler book Advanced Rails Recipes coming soon!

Ruby For Rails - (3/5) David A Black attempts to contextualise rails a little better within the Ruby ecosystem. Its a good read and if you are a new Rails developer you will soon learn that to a be a good Rails developer you have to have strong understanding of Ruby and that means some of Ruby's rather powerful and perhaps unfamilier features like meta programming. This books goes somewhere to help out here but I did find it somewhat disjointed and difficult to read in places.

A very important point is that there is a whole lot more to being a good Rails developer than just Ruby and Rails knowledge. Books to do with managing your projects, version control, javascript, XHTML, CSS, Linux, HTTP, web servers are all very important reads to becoming better Rails developers. So in a future post I'll be talking about the best non Ruby and Rails books for Rails developers.

Long live push_with_attributes

2007-01-12T19:40:00Z

So the general rule tends to be as soon as you have has_and_belongs_to_many associations and you want to use push_with_attributes then you are probably missing a join model. has_many :through has made it easy for us to promote our join tables to join models so we now have power in our joins. But although at a glance it looks simple you soon find that out of the box you loose a lot of functionality converting has_and_belongs_to_many associations to has_many :through associations. For a start we have no push_with_attributes anymore which was kind of the reason we went this route in the first place.

Well there's no need to panic. We can add our beloved push_with_attributes function back onto our association just like this...

  has_many :albums, :through => :artist do
    def push_with_attributes(album, join_attrs)
      Artist.with_scope(:create => join_attrs) { self << album}
    end
  end

Now adding this to all the necessary :through associations doesn't seem very dry so what we can do is use the power of Ruby meta programming and extend ActiveRecord's through functionality to include this out of the box.

Rolling with Mephisto

2007-01-06T10:51:00Z

Got Mephisto up and running on my site5 hosting thanks to a great comprehensive article from Raymond Lim. There are quite a few steps but following them carefully did the trick for me. Not many themes for Mephisto yet. There is this theme gallery but that is not an awful lot of choice. Perhaps I'll have to come up with one myself. I also couldn't upload themes through my Mepohisto admin section so I ended up just copying them manually to where there expected in the Mephisto directory tree and that did the trick.

Added a few nice sidebar widgets for photos, music and books. I love LibraryThing at the moment so I was pleased to get this widget tidied up with a bit of CSS and into the blog sidebar. Also I am loving monoslideshow. I'm a great believer in the use of flash for neat dynamic features on web pages that don't take over the page and monoslideshow makes it really easy to add these.