Archiving101.com; in depth no nonsense information about archiving and related technologies.
30th January 2008

Old data is toxic waste?

I ran into this quote on Hu Yoshida’s blog who is the CTO of Hitachi Data Systems on some of his predictions for 2008:

“Data more than 60 days old on production systems will be considered toxic waste. Structured data such as databases and semi-structured data such as e-mail and document management data are increasing dramatically as they are required to hold more data, longer, for compliance reasons. This will call for new types of archiving systems that can scale to petabytes and provide the ability to search for content across different modalities of data. “

Interesting .. I never thought about calling it ‘toxic waste’ before.

posted in storage | 3 Comments

30th January 2008

More about Journaling vs Log shipping (or other ways)

Still one of my favorite topics and seeing the discussions this triggers on the blog, also its high on other peoples lists.   While Googling for some information I ran across this prime piece of whitepaper. Quest’s Compliance Archiving with Microsoft Exchange Server.   Written by Michael Tweddle, who is the Technical Director for Intelligent Messaging Solutions at Quest Software this document initially looked interesting to read (I can recommend drinking this with a good whitepaper on a Saturday night).  

 Now .. it was all fine till I ran into the following:

“There are two key advantages of using journaling versus log shipping for e-mail compliance archiving: since log shipping is not designed for compliance, it does not capture any kind of recipient information in the To, CC, BCC fields, and distribution list expansion unless an organization manually retrieves this information.

It looks like Mr Tweddle does not understand the way Transaction Logs work in Exchange … after all how else would you be able to reconstruct an Exchange Database using transaction logs from an older backup?   For this Mr Tweddle .. I have to unfortunately award you the famous Lost Envelope Award.

posted in journaling, Lost Envelope Award, competition | 1 Comment

21st January 2008

The Raw Story: Email missing from Cheney’s office on day White House told to preserve documents in CIA leak

Source: http://rawstory.com/news/2007/Email_missing_on_day_White_House_0121.html

New report shows archives gone on several key days in Plame investigation

Among the sixteen days for which email are missing from Vice President Cheney’s office is Sept. 30, 2003, the same day the day the Justice Department and the Federal Bureau of Investigation announced they were investigating who outed former CIA officer Valerie Plame Wilson.

That morning, then-White House counsel Alberto Gonzales ordered the president and the vice president’s staff to “preserve all materials that might be relevant” to an inchoate Justice Department probe.

“We were informed last evening by the Department of Justice that it has opened an investigation into possible unauthorized disclosures concerning the identity of an undercover CIA employee,” Gonzales wrote in a terse Sep. 30, 2003 email. “The Department advised us that it will be sending a letter today instructing us to preserve all materials that might be relevant to its investigation. Its letter will provide more specific instructions on the materials in which it is interested, and we will communicate those instructions directly to you. In the meantime, you must preserve all materials that might in any way be related to the Department’s investigation.”

The analysis was released over the weekend by Citizens for Ethics and Responsibility in Washington (CREW), a D.C.-based ethics watchdog.

The White House said in a court filing last week that backup tapes, which contained archived copies of the e-mails, were recycled as part of a policy the White House had in place until October 2003.

Special Prosecutor and Chicago US Attorney Patrick Fitzgerald convicted Vice President Cheney’s former chief of staff I. Lewis “Scooter” Libby of obstructing justice and lying to investigators last year. Fitzgerald noted in a January 2006 letter that some of the White House’s emails had not been archived.

Emails gone on day Bush said he’d ‘take care of’ leaker

Ironically, Cheney’s office is missing emails from the very day President Bush told reporters he’d “take care of” whatever staff member had actually leaked the CIA agent’s name.

“If there is a leak out of my administration, I want to know who it is,” Bush said Sept. 30, 2003. “And if the person has violated the law, the person will be taken care of.”

The day before, then-White House press secretary Scott McClellan had said there was “nothing, absolutely nothing” to suggest any White House involvement.

“And that includes the vice president’s office, as well,” McClellan added.

Much remains to be learned about what happened to White House e-mails on 473 days for which they seem to have disappeared. A lawsuit brought by CREW and the National Security Archive and planned hearings from the House Oversight Committee are trying to find out just how much of the historical record of the Bush administration ended up in the White House recycling bin.

Cheney’s office also is missing e-mails from Oct. 4, 2003, when the Justice Department demanded that the White House turn over “all documents that relate in any way” to the leak of Plame’s identity. E-mails are also missing for the following day, during which the probe intensified and CIA director George Tenet found himself at the center of it, “caught between his loyalty to the president and defending an agency enraged” at Plame’s exposure, according to the New York Times.

As Fitzgerald’s probe continued over the next few years, emails continued to disappear, CREW says. More e-mails were missing from Cheney’s office on Feb. 16, 2005, when a court ordered reporters who had discussed Plame’s identity with administration officials to testify about those conversations.

All in all, some 473 days of emails are missing from various Administration departments, according to a House Democrat who saw a White House presentation on the files.

posted in eDiscovery | 0 Comments

18th January 2008

Governing.com: Delete at Your Own Risk

Source:  http://www.governing.com/articles/1email.htm

Governments need a system for managing their mountains of e-mail. Very few have one.

For several months last fall, the St. Louis media had a field day with Missouri Governor Matt Blunt’s office for doing the equivalent of crumpling up important office correspondence and tossing it away. Employees weren’t using a wastebasket, though. They were tossing out messages by clicking “delete” on their computers. Staff members insisted there was no written policy in their office on saving and deleting the e-mails. They said they routinely erased the messages because they didn’t view them as part of the public record.

Most other state offices were quick to disassociate themselves from this approach. The attorney general, the secretary of state and the auditor all announced that, unlike the governor, they treated e-mails as public records and retained them accordingly. Finally, in November, Blunt put an end to the controversy by announcing his own stringent new policy. A “permanent” e-mail retention system would be created, and employees would no longer be able to make case-by-case decisions on what to save. The governor gave his Office of Administration the task of developing technical systems to permanently save every single state government e-mail. He didn’t say how much that would cost or how much additional storage would be needed.

Missouri’s is not the only government that has been stumbling over vague or non-existent e-mail policy. Millions of state and local employees in jurisdictions all over the country correspond by e-mail every day without giving much thought to what should happen to the product. They may come to regret that behavior. Not only are records, and history, being lost, but many government lawsuits now turn on what is buried in old e-mail messages. Government policy simply has not kept up with the evolving technology. “At the moment,” according to Charles Davis, of the National Freedom of Information Coalition, “everyone is looking up and saying, ‘Maybe we ought to be keeping this stuff.’” But few have come up with clear rules governing where and how to keep it.

Some elected leaders are still trying to hold the line against long-term storage, but generally they aren’t succeeding. Last August, for example, Washington, D.C., Mayor Adrian Fenty issued an executive order directing employees to purge most e-mails after six months. Three months later, after resistance from the city council, he withdrew the order. The council felt six months wasn’t long enough to protect critical information and provide evidence against misconduct. “With the punch of a button,” one council member said, “many important, vital public records would have been lost.”

In general, though, it’s proving difficult for archivists and other information technology specialists to get across the message that government documents sent via e-mail can’t be viewed as ethereal missives. E-mail is simply another way to distribute a public record. As time goes on, courts are likely to hold governments increasingly responsible for organizing and saving those records in exactly the way they would save paper records. Logically, that would require them to set policies for computer use and offer training on how to properly handle and retain e-mails. Few do.

The whole problem is complicated by the fact that government workers use their office computers for a huge variety of purposes, many of them official but many personal or even questionable. Employees in state and local government use e-mail and the Internet to do their taxes, run eBay businesses, upload to YouTube and plan lunches and other personal events. If a government tolerates those uses, it seems a bit excessive to demand that every lunch invitation be saved and stored in perpetuity for public inspection.

Some states are beginning to deal with the problem by creating clearer rules for what employees can and can’t do with their computers on office time. Washington State has issued a formal directive stating that computers are for state business. Employees may use them for personal needs if they are not political or commercial. But the state has a right to monitor and preserve any message that is transmitted.

Texas has chosen to draw distinctions between categories of e-mail, carefully delineated in a “records retention schedule” one- and-a-half inches thick. “Transitory” e-mails deemed irrelevant to the conduct of government can be destroyed after a week. Others, including any sent to a constituent service office, are supposed to be kept “in perpetuity.”

Critics say this system is nothing but a giant loophole. “If it’s the governor,” says Charles Davis, “I would imagine all sorts of things would become transitory. That’s just a black hole.” Davis argues that in many cases, no one can be sure what records are important until long after they are created. The daily comings and goings documented by e-mail can be “terribly” important, he warns, if a scandal erupts involving a senior official. “Look, he’s hanging out with all these lobbyists,” Davis offers as an example. “How do we know that? The ‘transitory’ e-mails.”

In other places, elected leaders are devising ways to escape permanent retention of e-mails by sending them through private accounts, thus making it difficult or impossible even to find them. The chief aide to one Phoenix councilwoman transmitted a memo to staff members telling them to send e-mails on controversial issues to an AOL account rather than to the city’s e-mail system. City policy in Phoenix “encourages” employees to do city business on public accounts, rather than private ones, but does not require it. Hence, the council members were not breaking any rules. When a newspaper reporter asked for e-mails from seven members, including those sent from personal addresses, the members agreed to provide them. But a city spokesman concedes there was no way to know if all of the government-related e-mails from personal accounts were produced.

Another way for employees to avoid having their e-mail land on government servers is to send it via their PDAs by a method called “pinning.” Pinning allows two people to send messages back and forth directly to each other’s PDAs, without going through the government computer network. Few departments have policies prohibiting pinning, if they even know what it is. Information can be created and stored on PDAs, flash drives, hard drives and other pieces of equipment, and never reach the government system for storage.

One of the reasons governments aren’t managing e-mail well is that it doesn’t take up any physical space. If the amount of information now being clicked back and forth were still on paper, it would bulge out of filing cabinets. Agencies couldn’t let the piles stack too high. With the advent of electronic storage and search capability, there’s a strong inclination on the part of many governments not to do much of anything. If they need something, they assume they can find it. If someone deletes it, they aren’t inclined to worry. “We’re so busy generating and consuming and digesting information,” says Adam Jansen, the deputy archivist in Washington State, “there’s no time left in the day to manage it.”

PRESSURE FROM THE FEDS

In fact, though, if there was any doubt about the importance of public e-mail management, it should have disappeared in December 2006, with a change in the Federal Rules of Civil Procedure. Under those rules, state and local governments that become litigants in a federal case will have to produce any electronic information considered relevant to the case. If they can’t easily retrieve e-mails because they haven’t established an efficient way to store them, it’s going to cost a lot in staff time. Employees might have to review millions of e-mails to find which ones deal with the plaintiff. If they’ve deleted crucial e-mails that are public record, that creates other issues.

To guard against these problems, Washington State created a consolidated system for use in the discovery stage of lawsuits. Last September, the National Association of State Chief Information Officers issued its own report on preparing systems for e-discovery.

Many of these federal cases involve the use of government e-mail by one employee to harass another. For instance, an e-mail of a racist nature that is sent to everyone in an organization or passed along electronically. “Nasty, nasty jokes,” says Leonard Dietzen, a Tallahassee lawyer who represents governments in harassment cases. “They seem funny at the time. A year from now, when they’re in court, they’re not. It embarrasses us at trial and in the media.”

Before e-mail, it was hard to prove that an agency culture was oppressive. Plaintiffs had to show it through the word of other employees. Now it is set in digital stone. Lawyers can request e-mails with the plaintiff’s name on them, whether in the body of the e-mail or in the “to,” “from” or “cc” lines. Or ask for everything on the hard drive.

It might be tempting to conclude that the way to deal with this problem is simply to establish a policy of rapid deletion, thus eliminating the evidence. In fact, though, stored messages are as important to the defense in an employee-related case as they are to the plaintiff. They help governments prepare an argument and potentially save thousands of dollars in claims and back pay. If an employee sues, claiming sexual harassment by a superior, but investigators discover e-mails revealing he or she welcomed the advances, the government is in a stronger position in court.

For that reason, lawyers and archivists recommend that government agencies refrain from reassigning computers immediately after an employee is terminated. Instead, they should cull the information from that computer before a new user erases documents the government could use in court should it need to. There’s no telling when it might need to. Governments can be sued for years after an event. “You wouldn’t believe how many mistakes are made and just how obvious this is,” says Dietzen. “People finally are waking up.”

posted in compliance, eDiscovery | 0 Comments

18th January 2008

Email Archiving for Dummies

Not sure what category this would fall under, but I’m suprised no one else has come up with this idea years ago.   Recently released book ’Email Archiving for Dummies’ written by Mimosa Systems’ Bob Spurzem and Bill Tolson is now available.  You can get your free copy at the following site http://www.mimosasystems.com/Dummies/

posted in Uncategorized | 0 Comments

8th January 2008

Microsoft acquires FAST Technologies

For a nice 1.2 Billion USD.  

http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9056285&intsrc=hm_list

January 08, 2008 (Computerworld)Microsoft Corp. says that buying Fast Search & Transfer ASA, in a $1.2 billion deal announced today, will make it the only vendor that can offer a unified enterprise search platform capable of scaling to billions of documents.

But at least one analyst thinks that customers will face some risks as Microsoft tries to integrate its SharePoint content management software with the search technology offered by Oslo-based Fast Search & Transfer, which is known as FAST.

Tying together the two product lines “will be a challenge,” independent search analyst Stephen Arnold wrote today in a blog posting in which he also predicted more consolidation among the 50 or so companies now competing in the enterprise search market.

Arnold said he thinks that the engineers at Microsoft and FAST are up to the integration challenge. But, he added, “the question will be, ‘How long will the meshing take?’ If speedy, Microsoft can expand its service offering and put another hurdle in the path of companies like Google eager to win more of the Microsoft market. If slow, the delay will allow further incursions into Microsoft territory by Google as well as IBM, Oracle and SAP, among others.”

During a conference call after Microsoft announced its intentions to acquire FAST, Jeff Raikes, president of Microsoft’s business division, declined to comment in detail about the company’s product integration plans.

But Raikes contended that the combination of the two vendors will give Microsoft a leg up on its enterprise search rivals. “Most customers are recognizing that they don’t want their search to be fragmented across their organization,” he said. “They want to choose a search strategy and products from one vendor.”

Currently, corporate users have to “choose between a high-end-focused search [platform] or a broader infrastructure technology,” Raikes continued. He said that by marrying FAST’s products with SharePoint, “we are clearly the leader in end-to-end search in the corporation.”

FAST has about 2,000 customers, according to Arnold’s blog posting. The Norwegian company reported $162 million in revenue during 2006, but its revenue and stock price collapsed last year, forcing it to lay off one-fifth of its employees and slash unsuccessful product lines in order to regain profitability.

John Lervik, FAST’s CEO, said the restructuring refocused the company “on its search platform, which fits very well with Microsoft’s strategy.” FAST’s board of directors unanimously recommended that shareholders accept the buyout deal, which is expected to be completed in this year’s second quarter.

Raikes was undaunted by FAST’s financial struggles, saying that corporations are drowning in information overload and clamoring for search technologies such as the ones offered by FAST.

“I find it fascinating that you can get last night’s football scores in five seconds, yet it can take five hours to track down last year’s business plan,” he said. “We believe enterprise search will be for workers tomorrow what Internet search is for consumers today.”

Microsoft already offers several search products, including SharePoint, which the company says has about 85 million licensed end users.

“SharePoint can deal with tens of millions of documents,” Raikes said. In comparison, the latest beta releases of FAST’s search software can handle 1 billion records per server, Lervik noted.

In addition to the corporate plans, Lervik said that there have already been talks between the two companies about how to integrate FAST’s technology into Microsoft’s Windows Live search engine for home users.

FAST competes with specialized search vendors such as Autonomy PLC and Isys Search Software Pty., as well as with the likes of Google Inc. Sydney, Australia-based Isys was quick to raise questions about the planned acquisition by Microsoft, claiming in a statement that the combination of SharePoint and FAST’s software will create “the potential for real integration headaches” for customers.

(For more analysis of the Microsoft-FAST deal, see this blog posting on Computerworld’s Web site.)

posted in vendor selection, search | 0 Comments