The irony of terrorism and Enron/WorldCom-style financial disasters is that their terrible impact is forcing firms to radically re-think data management strategies in ways they should have done long ago - and which could also provide higher returns.

The financial collapses and geopolitical events of recent years have led to the introduction of numerous new laws, regulations and corporate governance responsibilities intended to dramatically overhaul the financial sector for the better.

These include Europe's Basel II, the US Sarbanes-Oxley Act and Patriot Act, the UK's Financial Services and Markets Act 2000 and anti-money laundering legislation.

These rules deal with operational risk, records management, visibility of process, business integration and reporting. They are forcing financial services firms to thoroughly reassess how they collect, use and store data.

A climate of improved risk management, operational transparency, financial reporting and corporate accountability is now the order of the day for national and international firms, with stringent penalties for those that fail to meet the challenge.

This is causing a rethink of how IT infrastructures can be used to capture, analyse and store data more effectively.

"If you manage to produce your management information in a much more standardised and accurate way much sooner after the year-end, then that is a tool that's going to make you more competitive, as well as allow you to meet the requirements of the new legislation," says Anita Bradshaw, senior industry expert at IT services company Computer Sciences (CSC).

"Effective use of data is increasingly becoming one of the key differentiators of a successful operation today."

But the change management process is a long and painful one. Legacy systems and processes have created silos of data which are hard to access to any great depth, often leaving valuable data undiscovered.

This does not sit well with the need for an integrated environment which allows a single global view of the customer, or the in-depth reporting requirements and deadlines set by legislation.

Many business processes, as well as systems, will have to be revised, integrated or even replaced, with all the corporate cultural complications that entails. This also means getting to grips with the pros and cons of a number of competing data management products and standards.

None of this comes cheap. The cost is likely to exceed that for year 2000 and euro projects, and even threatens the survival of less agile financial institutions, according to analyst Meta Group.

A typical large bank will need to assign £15m to £25m to IT-related compliance programmes. This is likely to be supplemented with additional programmes worth £30m to £100m between 2004 and 2007.

"The compliance challenge facing the industry is gigantic," observes Tony Gandy, research consultant at the Institute of Financial Services.

"With the current focus on risk measurement and data collection and analysis, regulation has become a very important issue for the IT department."

The knock-on effect in terms of emerging data management strategies has been dramatic. Companies are busy implementing technologies that allow a holistic view of customers, and of all the operations across the enterprise.

These include business intelligence, business process management, content and records management, email management, financial and accounting software, and enterprise resource software. All the big players in these fields are jumping on the bandwagon.

IBM, for example, recently unveiled 15 middleware packages, with applications such as risk and compliance management and claims processing.

EMC and Oracle are working together on a set of products and services to ease the storage of historical data requirements demanded by Basel II. And EMC, Iron Mountain and software startup KVS plan to release bundled archiving services aimed at compliance with US government mandates on email retention.

For compliance reporting needs, all the top 10 banks in Europe are using Cognos to report on risk positions, and assist in regulatory compliance.

"It allows us to add much-needed context around our data," says Patrick Hendrikx, head of front-end and reporting in financial services at UBS.

"We can quickly answer key business questions to determine our level of risk in a particular country or partner dimension, and quickly make decisions to protect ourselves. We can look for a high-level portfolio ceiling view, or a more detailed transaction-level analysis."

Smaller UK banks and building societies are being targeted with a Basel II managed service from Fujitsu Services, in partnership with Link, which provides transaction management services to the finance sector.

The idea is to help them get the same benefits from Basel II compliance as larger companies such as Barclays and Halifax.

By allowing them to share data in a secure and anonymous manner, smaller financial firms should be able to calculate more advanced risk measurement, reduce capital reserve requirements and improve competitive advantage.

Organising the data about your data
For most, the basic building blocks for compliance are understanding metadata (data about data), developing data standards, and corporate data modelling.

Legislation is encouraging all financial institutions to push the boundaries of how their data can be used. One household name in financial services, a customer of business intelligence consultancy Business & Decision, has invested in datamining, transporting large volumes of customer and account data from a mainframe marketing database to a new dedicated NT server.

Once the new environment was in place, more users saw the value in being able to access this rich source of customer information, and a 'corporate warehouse' was born. This data now has many uses, including a 'customer measures' project, which allows staff to determine who their key customers are.

In the IT and business world, knowledge starts with data, and only then do real revenues flow. The key issue for financial firms today is the gap between spending on data management systems for regulatory compliance, and turning the competitive knowledge they deliver into hard cash.

During this period of overwhelming change, financial services companies have no choice but to move in innovative ways on data management. The big question is how quickly they can act, and so steal competitive advantage in a volatile, fast-moving sector.