Retailers and wholesalers are laying the groundwork for a new breed of large-scale data bases designed to feed a vast array of decision-support initiatives.
Called data warehouses, these repositories contain historical information drawn from multiple, disparate sources and make it accessible to all levels of an organization to conduct analyses on an ad hoc basis.
As category managers and company presidents grow more adept with decision-support systems, however, their hunger for better information, more powerful tools and faster access also grows. The problem is, retailers told SN, this accelerating demand for processing power outstrips the capacity of existing systems, forcing them to find new solutions.
Retailers' relational data bases, already processing day-to-day transactional data, were never designed to support on-line analytical processing, information systems executives said. Data warehouses dedicated to decision support can handle the demand but issues of size and complexity are enough to intimidate some retailers from embarking on such a project.
"Data warehousing is a hot issue right now. Everyone is feeling pressured to have it ready but a lot of people are still behind, stuck in 'analysis-paralysis,' " said Cindy Cooper, management information systems manager of large systems at Pay Less Supermarkets, Anderson, Ind.
Pay Less' data warehousing project, like those undertaken by other retailers, was driven primarily by the desire to enhance category management programs and to sharpen merchandising strategies as new players enter the market.
A commitment to category management was the driving force behind the project under way at Salisbury, N.C.-based Food Lion, whose massive 500-gigabyte data warehouse may be the industry's largest.
"Our initial implementation was with category management. We saw that as our first opportunity and it whetted our appetite for what we thought data warehousing could do for us," said Ed Benner, vice president of information technology at Food Lion.
"It gives us a chance to provide deeper levels of understanding, deeper levels of analysis than we were capable of before," he added.
"Through data warehousing we are trying to dig out information that in the past may have been buried in one personal computer data base. We are trying to bring information to the surface and integrate it to help fuel organizational change," Benner said. "Then we can start driving information across all areas of the company through this enterprisewide data base."
Indeed, retailers who are serious about understanding product profitability on a granular level through category management -- and to get closer to their customers through micromarketing -- are erecting data warehouses.
About 44% of the 1,500 companies recently surveyed by Meta Group, Stamford, Conn., indicated they were involved in a data warehouse project.
"I think data warehousing is growing very fast in our industry," said Joe Ratay, corporate architect of information services at Giant Eagle, Pittsburgh.
The chain's 150-gigabyte data warehouse is used today to support executive information systems. Once the category management piece is added and customer shopper card data spanning 13 months is fully loaded, Ratay expects the data warehouse will grow to about 250 gigabytes.
The use of such mammoth open architecture data bases was virtually unheard of among supermarket retailers until recently. Two years ago, a 20-gigabyte open data base was something to boast about; larger systems were confined to a proprietary mainframe.
Today, decreasing hardware costs and improved software scalability have brought data warehousing within reach of more retailers.
"We are all riding the cost-performance curve of larger servers and the large relational data base manufacturers have products, so even a medium-sized retailer can get into data warehousing," Ratay said.
"It's a combination of wanting to take category management forward and the availability of technology," he added.
While decreasing systems cost may make the project easier to undertake, the complexity involved can be daunting.
"Building a skyscraper requires a little more planning up front," said one data warehousing expert. "The foundation is different. The materials are different. And when you get to the 50th floor, you can't spontaneously start to re-engineer the thing."
Such projects must be designed with "technical elegance" or they may risk failure, said Chris Horrocks, senior partner in data warehousing practice at Computer Sciences, Waltham, Mass., a division of CSC, El Segundo, Calif.
"One client I had made his first attempt with a 'Field of Dreams' approach -- like the baseball movie. He thought, 'If I build it, they will come.' Well, they did not come" because poor planning resulted in flawed design.
Selection of data to load into the warehouse is by far the most challenging task.
"There's an awful lot of data to collect," said Bob Heise, vice president and chief information officer at Richfood Holdings, Mechanicsville, Va. "The issue is, what data do you include? How do you decipher what data would be nice to have and what is required?"
Food Lion's Benner said selecting and collecting data from various data bases was a significant task.
"Historical data is not always perfect, so you have to make decisions: Do you want to take it as it is? Do you want to clean it up? Do you want to come up with new data by going back to some [original] source and rekeying it?" he said.
"All of these decisions have to be made in the context that everybody knows what we've got. We made it a joint analysis," Benner said.
"This is not just an IT project," he added. "This is an organizational project" that affects all areas of operations.
When laying the foundation for a data warehouse, retailers must grapple with other seemingly basic but often complex questions, such as how long particular types of data should remain in the system.
For tracking of product movement for promotional and profitability analysis, Pay Less stores two years' worth of information in its data warehouse.
Another retailer maintaining two years' historical data is H-E-B Grocery Co., San Antonio, whose year-old data warehouse has grown to at least 200 gigabytes. The chain's category managers access the information to make decisions on which products to put in which stores, how much product to send to the stores and the proper product mix.
Pay Less' Cooper said keeping data on product sales and movement for a two-year period has proven valuable in analyzing risks and opportunities when a new player enters a market.
When the retailer learned that a Meijer Inc. superstore was due to open nearby, the company was able to anticipate the potential effect on sales, by category and stockkeeping unit, based on data gathered when competitors entered other markets.
Key historical data that's available on a granular, SKU level not only sharpens the immediate decision but heightens users' confidence in decision-support tools overall, she said.
"They start believing in the data," Cooper said.
This shift of data analysis tasks from information systems staff to users at all levels of an organization frees the IS department to pursue other, more mission-critical projects, executives told SN. The users, meanwhile, bring their expertise to a higher level.
"We are trying to resurrect a 1960's phrase to 'Give power to the people,' " said Terry Ransom, executive vice president and chief administrative officer at Harry's Farmers Market, Roswell, Ga. To that end, the retailer built a data warehouse. However, cost and flexibility issues led to a shift in strategy. "We realized that data warehousing is not really a good idea for us, so we are moving to a three-tiered [client-server] architecture," he said.
"We are not scrapping the data warehouse altogether. We are redefining it," he said.
Ransom said he believes raw data should have a "life of its own" and that categorization of products, for example, should be conducted at the user level. Middleware, the center layer of client-server architecture, then retrieves the data from multiple sources in a manner that's transparent to the user.
Product-specific information stored in a data warehouse and locked into a particular category limited Harry's ability to reclassify certain products without losing the historical data. "Our categories may not be like the rest of the [supermarket] industry, which is one drawback," he acknowledged.
Ransom said time and money spent for periodic loading of the data was another liability.
"There is a huge amount of overhead related to taking data from disparate systems, or even lots of data from the same system, and putting it into a data warehouse format," he said.
"It's not a cheap thing to get into, so you've got to be sure there's value provided back to your company -- value that's going to be gained from these 'knowledge workers' who are using it," said Food Lion's Benner.
He emphasized the importance of developing "knowledge workers" -- or users of decision-support systems -- in tandem with building technology. "It's a twofold effort," Benner added.
