Deep_merge: Ruby Recursive Merging for Hashes 13

Posted by science on May 19, 2008

Welcome to “deep_merge” – a ruby lib to help merging complex hash structures.

Ruby provides some nice merge capabilities in hash and array. But it rightly doesn’t give us recursive merging, because it’s too poorly defined to standardize. However, recursive merging sometimes solves problems that can’t be solved other ways.

Take this code:

h1 = {:x => {:y => [4,5,6], :z => [7,8,9]}}
h2 = {:x => {:y => [1,2,3], :z => 'xyz'}}

If you want to merge these two hashes, what should happen? Well there are several possibilities. Let’s see how deep_merge handles it:

h2.deep_merge!(h1)
# results: h2 = {:x=>{:z=>[7, 8, 9], :y=>[1, 2, 3, 4, 5, 6]}}
# notice we overwrote 'xyz'! (That's what the bang means)
# Let's try it without the bang:
h1 = {:x => {:y => [4,5,6], :z => [7,8,9]}}
h2 = {:x => {:y => [1,2,3], :z => 'xyz'}}
h2.deep_merge(h1)
# results: h2 = {:x=>{:z=>"xyz", :y=>[1, 2, 3, 4, 5, 6]}}
# notice 'xyz' didn't get overwritten this time. No bang.

Let’s get a little more complicated with a “knockout” merge. We introduce a special string “–” which can “knockout” an element in an existing hash or array element:

h1 = {:x => {:y => ["d","e","--c"]}}
h2 = {:x => {:y => ["a","b","c"]}}
h2.ko_deep_merge!(h1)
# h2 = {:x=>{:y=>["a", "b", "d", "e"]}}
# notice no "c" any more!

Many of these features are configurable to your needs – feel free to read up in the source code. Home page and installation instructions are here: http://trac.misuse.org/science/wiki/DeepMerge

Trackbacks

Use this link to trackback from your own site.

Comments

Leave a response

  1. Sequence Mediaworks Blog » Cascading YAML config system? Wed, 04 Feb 2009 09:58:34 UTC

    [...] using a utility like Steve Midgley’s Deep Merge gem (Download) I can use common_bits as the base configuration and deep merge the individual site [...]

  2. Amy Wed, 04 Feb 2009 09:59:13 UTC

    Thanks! Your gem makes my new cascading config system work beautifully!

  3. Jónas Tryggvi Wed, 08 Apr 2009 09:33:00 UTC

    This looks like something that should be used in merging the hashes that happens in ActiveRecord, because if you chain many scopes together which each have different joins, it ends up doing duplicate joins cause the hashes aren’t identical.

    I ended up using strings instead of hashes for the joins to prevent multiple joins to same tables

  4. Nando Sola Tue, 19 May 2009 18:32:39 UTC

    Beautiful gem! Merging complex data from big forms has never been easier.

  5. Paul Hedderly Thu, 02 Jul 2009 02:07:47 UTC

    It’s just a shame that the priority rules are different to the default hash#merge

    a.merge b # items in b take priority

    a.deep_merge b # items in a take priority. Ugh

  6. Mathieu Sun, 11 Oct 2009 08:49:11 UTC

    Can you publish your work as a standard gem with http://gemcutter.org/ or http://github.com/technicalpickles/jeweler
    ?
    Without that, you work can’t be used in a simple and automatic way.

  7. Adam Jacob Sat, 14 Nov 2009 20:09:35 UTC

    Thanks for this – we’ve made it the mechanism by which Chef handles deep merges, and your knockout feature is truly epic.

  8. jtimberman Thu, 07 Jan 2010 13:47:53 UTC

    Hello!

    We’re using deep_merge in Chef, and I would like to package deep_merge for inclusion in Debian (and Ubuntu) as well. Debian uses an automated download of source, and doesn’t use gems or direct SVN checkouts, would you please post a tar.gz on the project’s rubyforge page so this can be retrieved in an automated fashion?

    Thank you for deep_merge!

  9. science Fri, 08 Jan 2010 10:32:18 UTC

    Paul – you can determine which takes precedence. If you issue “a.deep_merge! b” it will overwrite b. That seems like sensible behavior to me and is consistent with a lot of method calls in ruby, even though “a.merge b” for hash doesn’t behave this way (and the reason it doesn’t is because there is no “non-destructive” merge option in Hash. Since deep_merge adds the concept of non-destructive merge, it seemed worth changing the syntax. YMMV.)

  10. Matthew Kent Mon, 11 Jan 2010 10:30:14 UTC

    Think I’ve found an issue with merging booleans:

    http://gist.github.com/274454

    fix is here

    http://gist.github.com/274453

  11. Matthew Kent Mon, 11 Jan 2010 16:19:24 UTC

    Okay turns out there was a bit more to it as having extlib loaded changed the behaviour in overwriting unmergeables as they started calling blank?() :

    http://gist.github.com/274733

    it’s especially fun with FalseClass as blank?() always returns false so the value will never be overwritten.

    I propose something a more conservative default by making the use of blank?() optional:

    http://gist.github.com/274741

    to avoid any future confusion. Debugging this in a project that loads extlib for unrelated reasons was a bit time consuming :)

  12. science Sun, 17 Jan 2010 22:04:59 UTC

    Peculiar. Thanks for going into detail on this. Would you consider writing a solution in the code with a set of tests? If so, make sure the tests check to see if extlib is available when loading it, and don’t run the test unless it’s available (so the tests won’t crash on a non-extlib machine). Also, maybe write to STDERR with a warning that the extlib tests are being skipped in that case.

    I’m totally buried at the moment, but I’d be happy to upload a fixed version if you’re willing to code it. Thanks again for spotting it at least.

  13. Matthew Kent Mon, 18 Jan 2010 23:07:32 UTC

    Sure, how about this:

    http://magoazul.com/rubygem-deep_merge-0.1.0-merge_fixes.patch

    (be warned: sysadmins attempt at fixing :) )

    fyi: If your looking for someone to help manage the gem there’s some work being done here http://github.com/danielsdeleo/deep_merge (If he hasn’t already contacted you).

Comments