Winner of competition of CIO magazine for the best publication on key information technology for the corporate sector.

Incentive award
Ognian Shirkov: "The quality of data - a challenge requiring an adequate solution"

Assessment

Articles submitted for the competition were evaluated by an editorial board consisting of:

1. Nadia Krasteva - magazine editor in chief of CIO (Chair)
2. Ramona Chervenkova - CGEIT, a leading specialist in the field of management consulting and IT auditing, president of the Bulgarian section of the Association of Audit and Control Information Systems (ISACA).
3. Veselka Atanassova - Head of Information Security in Central Heating - Rousse JSC. Winner of the award for "efficiency" in the "IT Manager of the Year - 2008.
4. Marin Kalchev Director-General of Directorate General Information Systems, Institute. Winner of the award for "Management" in the "IT Manager of the Year - 2008.
5. Naiden Nedelchev - CGEIT, CISM, CEH, Head of Security Technologies Mobiltel EAD. Winner of the award for "Innovation" in the "IT Manager of the Year - 2008.
6. Dr. Pavlin Dobrev, technical director, ProSist Labs Ltd.

The editorial board assessment: Actuality, importance and usefulness of technology, methods and tools reviewed in this article, objectivity, competence, accessibility and attractiveness of the exhibition The ranking is based on the average of the individual assessments of participants in the editorial board.

DATA QUALITY - A CHALLENGE REQUIRING ADEQUATE SOLUTION

Integration and migration to a new information system represents a challenge for the IT departments of every company. Whether it is a data warehouse, ERP or CRM systems, this exercise is often undertaken to improve and optimize business processes, reduce costs and of course bigger profits. Not uncommon, however, such large investments of time, human and financial resources don’t have the expected return, regardless of the type or the provider of the system - the problem is obviously elsewhere.

"Garbage in - garbage out" - a free translation into the language of business means that even if you invest in the best software available on the market if the data that you feed in is not with a good level of quality, the results won’t be satisfactory, and reports can easily prove to be incorrect and misleading. Obviously a complete solution needed, to centrally analyze, evaluate and where possible to improve the quality of existing data and to prevent entry of data with low or poor quality in the information infrastructure of the company.

If all this sounds to you too technical and not that much related to the business, maybe it’s worth to think again. During the conference "Gartner's BI Summit", in March this year, the analyst from the leading research company Friedman Ted pointed out: "Despite lots of tools, big investments, many reports, and large data warehouses, without data quality decision-making is still largely a gamble. Data is…not only an IT problem. If you look at it [as only that], you're going to fail." To avoid such unpleasant scenario, the problem should be confronted with an adequate solution.

A solution

The large experience of Adastra Group in the data integration and data warehousing, is the foundation of creating a product for data quality - Purity.360. Established in 2003, the technology included in this decision is so successful, that it became the main driving force for the establishment in 2007 of the Adastra spin-off Ataccama Corporation.

Master Data Center (MDC), one of the two main product lines developed by Ataccama, is a specialized cutting-edge technology that delivers unified high quality information essential to your organization. This solution is fast and reliable, designed to integrate with existing data management systems, and processes millions of records across many data sources. Such cleansed, unified, and validated data is instantly available to wide range of enterprise applications such as, CRM, self-care systems, analytical applications and other internal systems.

Careful analysis of market trends and customer requirements in the early stages of product development identified key deficiencies that the existing DQ and MDM tools exposed. This allowed the tool to be designed from ground up to address these issues, resulting in a modern solution demonstrating strong competitive advantages. Based on JAVA technology, MDC works on any platform and with over 600 existing iWay adapters it can be easily integrated with virtually any product or infrastructure such as SOA. 

Figure 1: Integration of the MDC in the existing IT infrastructure

The product is designed with performance in mind and a it uses methods for parallel processing of the data, ensuring scalability and smooth growth of the processed volume, both working with existing large amounts of data and large number of online requests in real time. The latest version of the software due to optimization of the engine, is reaching speeds of analysis during data profiling in the order of millions of records per minute.

Product Features

MDC is developed as a “soft appliances”- it’s not dependent on any software or hardware of other providers. All necessary components are included with the product, from the application server, to the Web server, to transaction monitoring. The Ataccama technology only requires an operating system to operate.

One of the requirements of enterprise customers to software solutions for data quality is that they work in real time as part of their SOA architecture. MDC was created to handle large volumes of online transactions. The amount of records that is processed is in the order of tens to hundreds of millions, providing service level agreements (SLA) for less than a second for online services such as quality, identification and unification of the data.

Having foreseen the growing trend for Software as a Service, internal product architecture enables the implementation of all functionality as a service.

Localization and adaptation of the functionality to the local environment represents a challenge for a large part of, otherwise not bad products. The Ataccama technology is international from its creation. Moreover it is UNICODE based, which allows seamless operation with almost all existing alphabets and character sets, the solution is open and flexible, allowing the definition of custom business rules specific to each country, as well as providing an auto-configuration mode based on reference data. In addition to the flexibility of its product, Ataccama invests in research and analysis of reference databases for many countries, including Bulgaria.

The solution comes with a wide range of modules and algorithms for processing business data, pre-configured and ready for use, in accordance with the local requirements and business reality. Business modules are data models, business rules for quality assessment and consolidation of data, services and interfaces for particular business objects, such as individuals and legal entities, products, locations, addresses, contacts, vehicles, BULSTAT. In the product are contained more than 100 algorithms quality assessment, identification, exact matching and unification of the data. Algorithms can be combined in different ways with different evaluation approaches (scoring): deterministic, probabilistic, hierarchy based, business rule based etc.

When it comes to ease of use significant efforts has been made from a business point of view. Part of the package is MDC Portal - a web based application which is used by business analysts, managers, data stewards and data operators. The Portal provides full workflow monitoring concerning data management, allowing users to view and edit data, make changes in processes that require manual intervention, to review reports and more.

Thanks to the partnership with iWay and the variety of integration possibilities offered by them, MDC enables centralized, active monitoring and management of business processes in real time through the ability to determine the procedures and business rules, thresholds and automatic responses in the event their excess. The technology is a useful tool for thorough analysis and customer segmentation, classification of products, market research and to create custom data models used for data mining, risk management, fraud detection, market forecasts and more. MDC can be used to implement one of the prerequisites of Basel Capital Accord II implementation, especially regarding credit and behavioral scoring.

In conclusion

In many companies, the quality of the data is taken for granted. This clearly creates difficulties in convincing management to invest in such projects. Data quality is a real business challenge, not an exotic surplus. An adequate approach to the problem can lead not only to cost reduction and productivity increase of the overall IT infrastructure, but also to numerous business benefits, such as business process optimization and revenue increase.

Ognian Shirkov, IT Consultant, Adastara Bulgaria EOOD