I was catching up with a colleague recently and the conversation lead to some challenging projects that we worked on in Las Vegas nearly 15 years ago. At that time, Data Warehousing was a hot topic for many of the Las Vegas casinos, and they were throwing millions of dollars into these initiatives as they sought to attract and keep their high-value customers.
At that time, the most difficult tasks in building a data warehouse were:
- Connecting to and integrating the data from various systems: Casino and Hotel management systems, Retail, Spa, HR, etc. All had unique formats and some came from “black box” applications. There was a significant amount of custom coding involved, and a ton of trial and error.
- Standardizing the data: It was a mess! At that time, most of the data was hand-entered by hotel clerks, casino hosts, etc. Typos, misspellings, empty fields, garbage fields, notes in the Address 3 field. Again, more custom code and trial/error.
My colleague remarked that things got a lot easier once our company invested in a data integration platform. Less custom code, easier standardization processes, etc.
He went on to say that he’s still using some of those same tools today. But he’s experiencing some déjà vu. He feels like he’s regressing back into the realm of custom coding and trial/error.
His take: He feels like some of the data integration providers are coasting on the inertia of the past. They’re not putting a whole lot of effort into evolving their platforms to meet modern integration needs. There are significant gaps in their platforms, and he’s forced to fill these gaps by deploying (and supporting) custom code. We dug into some of specific gaps and came up with a “Top 10” list of signs and symptoms that indicate your data integration tools may be getting long in the tooth:
|Your Integration Tools may be obsolete if. . .||Description|
|Duplication of effort||Some older data integration tools only allow you to store job templates locally, making it difficult to share and re-use content from previous integration efforts. |
Modern data integration tools provide a repository of previous work/jobs that can be accessed and utilized to expedite new efforts. This saves you time and money.
|You struggle with Cloud Data||Older data integration tools were developed to connect with traditional “on premise” data sources. They perform well in this capacity, but things get complicated when you attempt to work with cloud or hybrid cloud data. |
Modern data integration tools provide native utilities and features that allow you to connect easily to any combination of on-premise, cloud, or hybrid cloud sources.
|You struggle with streaming data||Older data integration tools were designed to function in “batch” mode. Their marketing literature may pay lip service to “real time” or “streaming” abilities. But read the fine print and be prepared to invest in additional middleware and clusters of high-powered hardware to make that possible. |
Modern data integration tools are architected to process both batch and real-time data streams without additional middleware or custom coding.
|Increasing reliance on custom coding||Every data integration project is different, and invariably, a requirement that can’t be met by current tooling will surface. Older data integration tools are cash cows for their providers, and there isn’t a lot of new development or improvements being made to these platforms. |
As technology evolves more gaps emerge, and developers will find themselves writing more code. And fixing, testing, and maintaining code.
Modern data integration tools are actively supported by development teams – or are open source – and regular improvements are made (and new features are added) to minimize the need for custom coding.
|High cost of infrastructure & maintenance||Traditional data integration platforms typically require software maintenance, hardware, and dedicated administrative resources to keep the environment up and running. |
Modern data integration platforms are much less costly to maintain. In fact, those willing to consider cloud-based data integration subscriptions can enjoy savings of 75 – 90%.
|High licensing fees||Traditional data integration platforms are expensive. $100k will buy you enough licensing for a department-level implementation. Enterprise-level licensing can easily cost seven digits. |
Licensing for modern data integration tools is much more flexible and cost effective. For example, user-based monthly subscriptions – starting around $1,000 – are available.
|More middleware||Providers of older data integration tools will tell you that you can do ANYTHING with their tools. And they’re not wrong. You’ll just have to buy another piece of middleware. |
As technology evolves, they’ll crank out more middleware and the cycle continues. They pray to the god of sunk costs that you’ll stay on board.
Modern data integration tools – especially cloud and subscription-based solutions – realize they need to earn your business. Their solutions will evolve to meet current needs and trends in the market. And if they don’t, a competitor will.
|Slow development and test cycles||Ever tried cutting wood with a dull saw? It’s time consuming and dangerous. |
Same situation with aging data integration tools. You plod through the development cycle, fill in the gaps with custom code, test, re-code, and test again. Then go live with a sense of dread.
Modern data integration tools are a sharpened saw with utilities that allow you to “measure twice and cut once”, which can help you reduce development time significantly.
|Unintuitive and Byzantine User Experience||Older data integration tools are often a conglomeration of technologies that have been welded together over the years. This results in an unintuitive, inconsistent user experience with a learning curve measured in years. |
By necessity, modern data integration tools have a much more streamlined and intuitive user experience. These providers understand that:
(a) Simplicity and speed are key to their clients.
(b)Less-technical analyst roles are playing a larger role in integration efforts.
Modern tools aim to provide an intuitive user experience with a quick learning curve.
|Poor Documentation & Support||The documentation and support resources provided by the makers of older tools are often limited, confusing and vague. |
This is intentional – they have large consulting practices and want you to rely on them to make the software go.
Modern data integration providers want to make it easy for you to use their tools. If it’s easy and useful, you’ll continue to pay your subscription.
Their documentation is typically clear and contains links to numerous resources if one wants to dig deeper.
Many modern providers host user groups and online communities that encourage sharing of information, tips, training, etc. among users.
Are there any other gaps, deficiencies, or frustrations you’ve experienced? What features would you like to see in modern data integration platforms? What platforms are you using?
We’d love to hear your thoughts. . .And if you’re looking to modernize your data integration platform, reach out to us and we’ll be happy to make some recommendations.
(Hint: Here’s one)