Integration: Web Services

We've been looking at ways to share customer information between two applications:

How do we share customer data between these web apps?

One approach was having both applications use the same OLTP database. This presented some challenges; namely, it coupled the two applications very closely together, creating a huge ripple effect if either application needed to change. A second solution was to use ETL scripts to shift data between application databases. This decouples the applications a little, but integrating at the data layer means we lose a lot of context.

Web Services

For our third solution, let's explore the use of web services. In this solution, the Web Store would "own" the customer information. The Marketing application would store its own application-specific data, but it would make a web service call to fetch customer information. The solution might look like this:

Here, the Web Store exposes a service for Marketing to access customer information

The web service could be implemented many ways:

  • An RPC endpoint, using SOAP for operations like fetching and updating customers, perhaps implemented using WCF
  • A RESTful endpoint, with Customers as a true resource, exposed as XML or Json
  • A URL that just returns a CSV of Customers

Advantages

In the previous approaches, the Web Store application gave up control over its data. ETL scripts might have meddled with customer data, bypassing the application domain logic.

With this approach, Web Store retains complete control over how other applications access and modify the data it owns. It can validate updates to customers, reuse some of the domain model code, block updates to archived customers, and so on. Best of all, it can change the database schema completely without upsetting grumpy DBA's ;-)

Disadvantages

While this approach has many advantages over the previous solutions, it has a major downside: the reliability of the Marketing solution is coupled to the Web Store solution.

Although a critical system like Web Store is unlikely to be completely offline for a period of time, this architectural mistake could manifest itself in other ways:

  • If the Web Store is exceptionally busy, the Marketing solution may run very slowly
  • A bug in the Marketing solution (like calling a service in a tight loop) could have negative impacts on Web Store's reliability
  • If multiple applications begin to depend on Web Store's web services, the Web Store team may have to deal with a myriad of versioning issues.

These issues are especially important if either application has any kind of SLA.

Conclusion

It's important to note that while WSDL/MEX and technologies like WCF do a good job of decoupling applications by using contracts, they alone don't fully decouple the uptime, reliability and performance issues that come about when integrating applications.

What experiences, good or bad, have you had creating web services that are consumed by other applications (not just between client/server apps) in your enterprise?

A picture of me

Welcome, my name is Paul Stovell. I live in Brisbane and work on Octopus Deploy, an automated deployment tool for .NET applications.

Prior to founding Octopus Deploy, I worked for an investment bank in London building WPF applications, and before that I worked for Readify, an Australian .NET consulting firm. I also worked on a number of open source projects and was an active user group presenter. I was a Microsoft MVP for WPF from 2006 to 2013.

26 Apr 2011

If the Web Store is exceptionally busy, the Marketing solution may run very slowly -> Use CQRS

A bug in the Marketing solution (like calling a service in a tight loop) could have negative impacts on Web Store's reliability -> Use CQRS

If multiple applications begin to depend on Web Store's web services, the Web Store team may have to deal with a myriad of versioning issues. -> If you want to go public you must support backward compatibility

26 Apr 2011

@MilanChe, interesting direction. I wonder if CQRS is perhaps an architectural style on top of these integration patterns - arguably, you could implement CQRS on top of an ETL solution, or on top of a Web Services solution, or on top of a Service Bus solution (but probably not on top of a shared database). Do you think that's a fair assessment?

26 Apr 2011

Good write-up. Love your points about WCF (and other similar RPC-like frameworks) not decoupling from reliability and performance issues.

You can try to alleviate in some degree the impact using some decoupling techniques on initiator's side (circuit breakers, timeouts and so on) and in main system (some guard mechanisms that will be watching for performance degradation) but it's an additional complexity and you still get two coupled systems.

Will you go over other integration options?

26 Apr 2011

Yes, I agree with you Paul

26 Apr 2011

@Valeriu, thanks. Nice points on using some of the Release It patterns to improve the stability of both applications if you have to take this route.

Next up I plan to cover messaging using a service bus. Were there other solutions you'd like to explore?

Valeriu Caraulean
Valeriu Caraulean
26 Apr 2011

@Paul, if you will cover integration using some kind of Messaging mechanism you're done covering essentially different options.

And if you're looking for some serious work about integration the Enterprise Integration Patterns definitely shouldn't be forgotten.

Fernando Adachi
Fernando Adachi
28 Apr 2011

Paul, going to answer here your question from Shared db scenario...

Q: Is that what you had in mind, or were you imagining a separate service layer that would also be used by Web Store?

A: I was imagining a service layer. When more and more apps comes to use the customer data, it's not the responsability of the Web Store to keep the control of that data.

For the current scenario of Web Services, it is good if in the picture only the Web Store and Marketing App. If any plans to build or bring more applications to user/consume customer data, then a separete service layer uniquely responsible for the customer model should be created. It's a matter of responsability.

Also, with a "Customer Service Layer", all the disadvantages mentioned above would be eliminated. Sure we'd have to keep one more application but since the company is getting bigger, it won't be any headache.