WS-Deathstar for the REST of Us: A Story of Ruby, WSDL, and Salesforce

May 5, 2008

UPDATE: (Jan. 13, 2011) As so often happens in the Ruby world, this blog entry is out of date. I’ve heard reports about various SOAP libraries no longer working with Ruby 1.9, but I haven’t had the need to test them out. If you’re working with Salesforce and Ruby, development dm-salesforce seems to have carried on, meaning that you won’t have to do most of these steps.

I’m going to leave this up for posterity and to help anyone who may need to dig around in the undocumented parts of any of these libraries. If you are one of those unlucky people, please understand that you have my condolences and best wishes for success.


Sooner or later it’ll probably happen to you: WSDL. SOAP. WS-Deathstar.

The horror.

You say, “I use Ruby! We’re all RESTful around here, man! My webapp consumes and produces YAML and JSON with no angle-bracket residue (it’s better for the environment). Our Kool-Aid’s delicious!”

I know — it really, really is.

A long time ago (yesterday), I thought I was safe. Then I had to integrate an internal webapp with Salesforce.

Me: “Do they have a way to do feeds? JSON? Anything?”

Internet: “SOAP.”

Me: “That’s for those Java and C# people! Not hip cats like me.”

Internet: “Try our SOAP. It’s delicious.”

It really, really wasn’t.

Someone asked me, “Why not use Activesalesforce?” Sure, some of these problems could have been avoided that way. However, my app doesn’t use Rails or ActiveRecord. I figured I was going enough against the grain that I should just strike out on my own. And WSDL makes it easy (or just possible), right?

So, as half cautionary tale / half survival guide, I give to you my tale of SOA-WOE. Presented below are the steps I went through to get Ruby and Salesforce’s WSDL to play along… well, not nicely, but at least… well, let’s just leave it at play.

…And before I get into it, let me preface by saying that a lot of my problems look silly now. The answer are, of course, in the documentation. But separating the wheat from the chaff requires knowledge that most people don’t have when they’re starting out tabula ras(o)a.

By the way, if you’re reading this for generic Ruby/WSDL and not Salesforce information, you can skim the first three steps.

1. Signup

As all hip web developers have, I’ve read Getting Real. So I learned that you’re supposed to lower the amount of effort it takes for a user to get on board. Having to sign up just to see the API docs and wiki is not in that vein. It’s sort of the first signpost on the road that we’re treading into unfamiliar territory. Well, I guess them’s the rules.

2. Get your Security Token

So yeah, here’s another thing that is not necessarily straightforward, but I guess there’s a reason in this case: you have to have a Security Token to append to your password when logging in through SOAP. Overall, it’s probably a good idea.

Didn’t notice that? Neither did I. But don’t feel so bad, neither did this guy.

To get your Security Token, go to the Setup link at the top of the page, find the “My Personal Information” section and click on the “Reset your security token” link. It’s around there somewhere.

3. Get your WSDL

Now look on the left column under “App Setup”. Click “Develop” and find the “Download your organization-specific WSDL” link.

Look at all the choices! Well, which one should you use?

You could do what I did:

  1. Stare at the options for a little while
  2. Say “Awh, fuck it.” at a volume that may or may not have been heard by my fellow cube-sters (it probably was).
  3. Download “Enterprise WSDL”. Because I’m feeling enterprisey.

4. Install the SOAP4R gem

From what I’ve gleaned, SOAP4R is the same as the SOAP standard library. By why not get up to date with the latest and greatest:

gem install soap4r

5. Run wsdl2ruby

The example linked above builds the SOAP driver at runtime, which can take awhile. His method is good for playing around, but you should probably just go ahead an build them.

wsdl2ruby.rb --wsdl ~/Desktop/enterprise.wsdl.xml --type client

Change your paths accordingly. This will produce 4 files in your current working directory: default.rb (which does something), defaultDriver.rb (which does something else), defaultMappingRegistry.rb (which also does something, but not the same thing as the other somethings), and SforceServiceClient.rb (which does nothing).

So you’ve got these files. Don’t edit them! Someday you will run that code generator again and you’ll come to rue it. Some even say don’t version control them. Instead they’d put the wsdl under version control and add a Rake task to run wsdl2ruby (this assumes that wsdl2ruby won’t break in the future and it installed on any future machines). Choose for yourself.

Are we there yet?

No, but it feels like we almost are, right? I was once like you.

Let me first give you some incomplete sample code:


require 'rubygems'
gem 'soap4r'                                     # require 'soap4r' doesn't work
require 'soap/soap'
require 'defaultDriver'                        # this is one of the generated files that does something.

# d = driver;  I used one letter names here because I'm playing around in IRB.
d = Soap.new
d.wiredump_dev = STDOUT

# l = login result
l = d.login(:username => your_username, :password => your_password + security_token)

d.getUserInfo("")

Should work, right?

Nope.

6. Reset the Endpoint URL

The first error this code will produce is this:


SOAP::FaultError: UNKNOWN_EXCEPTION: Destination URL not reset. The URL returned from login must be set in the SforceService

This is simple enough to fix: after the login result (“l”) is returned, add d.endpoint_url = l.result.serverUrl

7. Maintain the Session ID

Now here comes the really fun part. The next error this code will produce is:


SOAP::FaultError: INVALID_SESSION_ID: Invalid Session ID found in SessionHeader: Illegal Session

Most of the search results in the old Google say this has something to do with the “Lock sessions to the IP address from which they originate” box being checked (which is located under “Administration Setup” -> “Session Settings”). This might be your problem, but in my case it was just that I didn’t know how to set headers in SOAP4R.

SOAP4R uses “header handlers” to modify outgoing and read incoming headers (and note that these are SOAP headers, not HTTP headers), so you’ll have to write one of your own (syntax highlighted version):


require 'soap/header/simplehandler'

class ClientAuthHeaderHandler < SOAP::Header::SimpleHandler 
  SessionHeader = XSD::QName.new("rn:enterprise.soap.sforce.com", "SessionHeader") 

  attr_accessor :sessionid 
  def initialize 
    super(SessionHeader) 
    @sessionid = nil 
  end 
    
  def on_simple_outbound 
    if @sessionid 
      {"sessionId" => @sessionid}
    end 
  end

  def on_simple_inbound(my_header, mustunderstand) 
    @sessionid = my_header["sessionid"] 
  end 
end

I found this thread and and this presentation helpful in figuring this out.

Pulling it all together, here’s the finished product (syntax highlighted version):

 
require 'rubygems'
gem 'soap4r'
require 'soap/soap'
require 'defaultDriver'
require 'client_auth_header_handler'

d = Soap.new
d.wiredump_dev = STDOUT

h = ClientAuthHeaderHandler.new          # Create a new handler

l = d.login(:username => your_username, :password => your_password + security_token)

d.endpoint_url = l.result.serverUrl         # Change the endpoint to what login tells us it should be
h.sessionid = l.result.sessionId              # Tell the header handler what the session id is
d.headerhandler << h                          # Add the header handler to the Array of headerhandlers

d.getUserInfo("")                                 # Make an API call; the empty String appears to be necessary

defaultDriver and client_auth_header_handler (the code from above) are assumed to be in the current directory.

More API calls can be found in the API docs.

Lastly, I want to point out another resource that helped out with this process that I didn’t find a good place to link elsewhere in the article: Getting Started with SOAP4R, by Mark Thomas.

Conclusion

Ok, so a lot of these problems are because I was impatient to get started. There were API documents with Java examples I should have been reading! Sorry, I thought we were working on the Web here, where HTTP is king. I know HTTP. I know a couple of serialization formats. I should be good to go, right?

If you’re a seasoned SOA-type then a lot of what I’m complaining about probably seems like old hat to you. Sure, I’m whiny. Kids like me don’t know how good they have it!

Still, I can’t help thinking that there’s no reason for the extra overhead that comes with WSDL and SOAP. It might be that I work with dynamic languages most of the time. I miss my REST. I miss my YAML and JSON. I wanna go home.

26 Responses to “WS-Deathstar for the REST of Us: A Story of Ruby, WSDL, and Salesforce”

  1. Great read. I love these kind of walkthrough a problem posts. It’s a nice diary of how difficult/easy the learning curve is on some of these projects.

  2. Andrew said

    @Walter: thanks for the comment. I’m glad you enjoyed it inversely to my enjoyment of doing the research for it. :)

  3. Morten said

    Hey,

    You don’t actually need the security token as long as you authorize the IP your application is going to be connecting from. I think it’s Setup -> Security Controls -> Network Access in Salesforce.

    /M

  4. Andrew said

    @Morten: Ah yes, you’re right. Thanks. I think a lot of developers are going to want to use the security token since they might be working on a machine with a dynamic IP. Good point though.

    -Andrew

  5. Wrighty said

    I feel your pain.

    I’ve spent this week looking deep into the bowels of SOAP for integration with Amazon’s Merchant@ program. Since discovering that the endpoint in the documentation (ending in .com) isn’t the right one (should have used .co.uk) I’ve been wading through the SOAP4R source to try and get SOAP with Attachements (SwA) working with Amazon’s particular take on it.

    I ended up creating a SOAP::Filter::Handler to deal with adding and extracting attachments from requests and responses. (And editing those generated files to boot.)

    The problems really boil down to a lack of documentation on both the service provider and the library. Luckily the source to SOAP4R is available with (somewhat simplistic) examples include to get you decent search terms to key into the library proper.

    It does feel a lot like wasted effort though.

  6. Sorry, guys, but your gripes are misplaced.

    SOAP itself is a fine protocol and in many ways, it’s SUPERIOR to RESTful JSON/XML. With a nicely defined WSDL, you’ll know that property “foo” exists in the message, that it’s a a String, and that’s 40 characters long. There is no Schema-Definition equivalent in REST, and you need a pre-arranged agreement of what will be exchanged across a RESTful endpoint. (JSON, XML, image-file data, etc.)

    Your frustrations come from the fact that WSDL2Ruby is not capable of gleaning a WSDL for these schema cues, and is the root cause of all your pain.

    For other platforms like .NET and Java, there are fantastic tools (WSDL.exe, WSDL2Java) that generate complete classes off of the WSDL; and it works beautifully.

    I had blogged about the shortcomings of WSDL2Ruby here: http://blog.lalee.net/blog/articles/2007/01/19/ruby-rails-and-soap4r

    Don’t knock the protocol just because you don’t have a good tool for your preferred platform.

  7. Andrew said

    @Laurence: You say my frustrations come from the fact that my tool can’t generate fully functional code from a WSDL description.

    Well, yeah. That’s true. But it wouldn’t in Java or .Net either with their “better” tools.

    Look at my two biggest “gripes”: 1) resetting endpoint URL and 2) session ID. How does WSDL generate code to handle this? It doesn’t. If it did, Salesforce’s “Quick Start” wouldn’t have us manually changing the endpoint and setting up session handlers.

    How would a RESTful client handle this? By using HTTP. I’d be redirected to the new endpoint (if that was even necessary). The session id would be a standard cookie (although there are RESTfuller ways of doing this). Hundreds of millions of user agents do exactly this everyday when they check their Gmail, buy things from Amazon, or do pretty much with authenticating a user. That’s more users than Salesforce’s WS API is ever going to have.

    So, after I’ve set up the appropriate credentials, I can GET /AndrewO/contacts.json. I can do this from a commandline or from pretty much any programming language with minimal library support.

    … and with no code generation. It’s not necessary.

    Because we have a simple interface. Four methods for interface is all we need (some say two).

    … and documentation. But less of it. Because we don’t have to explain the interface — the HTTP libraries do that for us.

    Ok, so with interface issues dealt with, what about data schema? No one says that you can’t use XML Schema, Relax NG or any other validation on a RESTful message. But here’s why you don’t see that as often in RESTful services: valid data starts with the developer. You have to _tell_ the programmer what everything means. There’s no strong AI that’s going to figure out what the symbols mean on their own.

    Clients that adapt to API changes are both undesirable and a pipedream. Data schema changes have repercussions for both the client and the server and the developers are going to have to be involved whenever that happens.

    … and that point even ignores the argument that you can’t completely represent all of the data rules using any of the popular SOA schema. What about data constraints that have nothing to do with the message, but with the environment? “Field X must be unique across the application.” “Return Y if some other external service is inaccessible.” “Only change this between 12am and 3am (the witching hour).”

    But somehow these constraints continue to be enforce. How? Because they can all be programmed into the application.

    Now, here’s the kicker: *The only valid description of an application is the application itself.* Everything else is best-efforts. Some applications are easier to read thanks to their host languages, libraries, and architecture (but that’s another discussion altogether).

    If we focus on making our (flawed) descriptions human focused (simple interface, easy serialization formats), we’re going to get better software than if it were written by machines (code generation).

  8. Andrew said

    @Laurence: BTW, I meant to start off with this, but forgot to add it: thanks for commenting. This entire article stems from the fact that the Ruby community is pretty monolithic when it comes to ROA vs. SOA. Although I disagree adamantly, I’m glad the contrary point of view here.

  9. Vipin Vellakkat said

    Hi All,

    I have a different type of problem.

    One of our client had created a WSDL in ASP and sent it over asking us to implement the same structure in ruby. I need to write a server app using these WSDLs.

    wsdl2ruby.rb –wsdl /home/Test.wsdl –type server –force

    using this command i got 4 ruby files such as… default.rb, defaultServant.rb, defaultMappingRegistry.rb, TestService.rb

    using these files how can i write a webservice server applicaiton?

    Thanks in advance,

    Vipin

  10. The style of writing is very familiar to me. Have you written guest posts for other blogs?

  11. doug livesey said

    THANKyou! I’ve been struggling with this for hours!
    I hate SOAP so much you can call me a hippy, but this has made the unbearable workable.
    BTW — ActiveSalesforce is currently broken, so that isn’t an option. It returns an error for the first call to each kind of object that you have to swallow with a rescue, thus requiring additional calls to the slow API.
    Thanks again,
    Doug.

    • Andrew said

      @Doug: Thanks! Glad to see that after this long, there’s still something useful to be had. Just, FYI: I’ve been using DM-Salesforce (available on Github) in a recent project and it’s going reasonably well. I’ve had to pull some hair to get a couple of the API calls to work, but the standard CRUD stuff works fine and is probably worth checking out if you’re able to use DataMapper.

      Good luck!

  12. [...] the Betfair API. Problem is, given the general disdain of the open source community for the whole WS-clusterfuck, there aren’t really any good SOAP libraries for dynamic languages around. I had a bad [...]

  13. Fuse said

    Thank you SO MUCH for this article!

    This is EXACTLY what I needed to get started on Ruby->Salesforce API working correctly.

    I was starting to pull my hair out trying to figure out this Session ID stuff!

    Fuse

  14. Kaushik said

    Hi,

    I’m trying to get data from sfdc. Thank you very much for the post. I’m currently using ‘retrieve’ to get information.

    Ex:
    soap.add_method(‘retrieve’,’fieldList’,’sObjectType’,’ids’);

    Is there a way we can get all the objects from sfdc and store them in an array ?

    Regards,
    Kaushik

  15. But we have a problem soap4r is broken under Ruby 1.9 and not updated since 2007, it seems dead :-/

    And all the other kids like handsoap or savon want to have much more and attention, and handwritten boilerplate code :-/

    • Andrew said

      Uh-oh… Looks like this is going out of date. I guess progress is great until it leaves you behind.

      The last SF integration project I did, I used dm-salesforce which seemed to use most of the same strategies as I did in this article. However, I believe it was based on the SOAP4R library as well, so it looks like we’re out of luck there. I’ll keep you posted if I come up with anything, but for now I’m sorry to say I don’t have a good answer for you.

  16. Dave said

    I had problems getting inbound headers to work as expected with the above code.

    As a workaround, I used the following client_auth_header_handler.rb for reading/storing/sending my auth “Ticket” (or could be sessionid) in the headers of each soap call.. note the different method name + args used on the inbound callback:

    client_auth_header_handler.rb
    ~~~snip~~~
    require ‘soap/header/simplehandler’

    class ClientAuthHeaderHandler @ticket }
    end
    end

    def on_inbound_headeritem(soap_header, my_header)
    @ticket = my_header.element["Ticket"].data
    end

    end
    ~~~snip~~~

    • Dave said

      Hmm.. didn’t paste right. Must have been the angle brackets.. trying again:

      ~~~snip~~~
      require ‘soap/header/simplehandler’

      class ClientAuthHeaderHandler < SOAP::Header::SimpleHandler
      TicketHeader = XSD::QName.new(“http://localhost/SomeRDCService/”, “TicketHeader”) # Trailing slash on url was important in my case
      attr_accessor :ticket

      def initialize
      super(TicketHeader)
      @ticket = nil
      end

      def on_simple_outbound
      if @ticket
      { “Ticket” => @ticket }
      end
      end

      def on_inbound_headeritem(soap_header, my_header)
      @ticket = my_header.element["Ticket"].data
      end

      end

  17. Dimitar Panayotov said

    While doing some generic Ruby WS research (needed to quickly test if a certain WS is delivering what the company needs), I stumbled upon this article and loved it, both the details and the style. Since then I have been trying soap4r, savon, handsoap. None of them impressed me, which kinda sucks because I love Ruby. :/

    My concrete problem was the WS-level security — but I won’t go into details. The WSDL itself hat to be downloaded from an SSL location, which I did successfully, but then the WS-level security basically killed my desire to move on.

    In short: have any of you guys have any update on a decent Ruby gem (or native library even, since one can always pack a gem which depends on one) which does support both Ruby 1.8 and 1.9 and delivers not only the basic SOAP stuff, but also has some coverage for the WS security?

  18. Jacob said

    This post saved me tons of time. Someday, I should probably contribute back to the community about my development struggles, but until then I’ll continue learning from awesome people like you.

    Thanks.

    • Jacob said

      I’m replying to myself because this is important.
      USE dm-salesforce

      (sudo) gem install dm-salesforce
      datamapper will return your life from the hell that is SOAP

      • Andrew said

        True indeed. Sorry if I caused you to waste some time. I added a block to the top of the article so people know how old it is.

        IIRC, I had moved over to dm-salesforce at some point after writing this. I had some problems with making API calls that didn’t map to Datamapper’s adapter… I think getting the changed records, but the specifics escape me. I ended up having to add some hacky code that dug down into the SOAP interface and called other methods. I tell you, it’s inescapable sometimes…

        But you’re right: dm-salesforce is a valid option and probably fits the majority of use-cases best.

      • Jacob said

        No apology needed, this article is still totally relevant. I walked through it and was able to get a good soap connection to salesforce, with a couple modifications to the files that were generated by wsdl2ruby.

        I didn’t notice your references to DM until a day later. DM is not that it’s a complete necessity for salesforce api, but it is definitely very nice to have such a well developed solution that we can just add in. It makes querying SF much easier and rails-like.

        Thanks very much for the article. I was about to give up on salesforce until I found it.

  19. [...] Hi All, I am not a Ruby guy, but last few days i had to look into this language, moreover in the context of calling apex web service method using Ruby. I love doing this using java and php, even i did in python also, but this time its a challenge for me to achieve this. Its a challenge because lack of documentation and sample codes on net and as a newbie in Ruby world. As every new developer i also searched many times on net but was struggling sometimes setting up Ruby, sometimes rubygems, sometimes things were not working. I left many times work in middle long before also. But searching and searching and i found this blogpost by Andrew very helpful. http://hideoustriumph.wordpress.com/2008/05/05/ws-deathstar-for-the-rest-of-us/. [...]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: