Archiving101.com

Hopefully this will trigger a little bit of a discussion.  I spend quite some time also reading international laws and regulations to find out which ones are applicable to the industry that I’m in.  By no means I’m any sort of lawyer, but I merely do this out of my own curiosity and interest.

If you have been in this industry or work for a large organization that uses archiving software then you probably are aware of the more common regulations that you hear; SEC 17-a, NASD3010/3110, Patriot Act and the list goes on.

However if you are an international business then you will eventually deal with the privacy laws in Europe or other continents which could result in some very interesting cases.  For instance take the case of the following document written by Fred H. Cate:

Legal Restrictions to Transborder Data Flows to Prevent Government Access to Personal Data

Now I don’t expect anyone to read the entire document however the Executive Summary lines out the problem that I see starting to occur.  In this case it involves personal data (but it could also be confidential data from lets say a chemical company) from Canada.  With the current regulations that are out there, it ends up that Canadian computerdata (even though it is considered confidential), can be surrendered to the US Government when it is stored on a physical system that is on US soil.   It opens up an entire new can of worms .. who technically owns the data if it is submitted through the wire .. what if a rogue country simply captures it and creates their own law stating that any data on their soil becomes public domain?

The writer makes a very good point in the fact that we are in urgent need of a multinational approach for protecting global information flows:

“The critical issues highlighted by the British Columbia report and law, and especially the question about how to deal with divergent national legal systems that increasingly come into conflict as data move across national borders, warrant thoughtful consideration. While the concerns are not new, powerful information technologies, global networks, and the multinational commerce, outsourcing, and information sharing they have made possible are inevitably going to cause new and more frequent conflicts between divergent national (and provincial) approaches to privacy and information management. Those same technologies and activities, our growing reliance on them, and the important values they implicate—including privacy and security—also heighten the urgency of finding multinational, diplomatic solutions that protect global information flows.”

I ran into this article, and while its close to a year old its content is still very relevant.   A blurb:

Do you know your backup from your archives? A slew of recent vendor-sponsored surveys shed doubt on whether you do — or if you do, whether you’re actually doing either task correctly.

A survey by Enterprise Content Management Association of the Association for Information and Image Management (AIIM), indicates storage consumers are basically ignorant when it comes to archiving. “Most organizations confuse email backup — being able to reconstruct a system from a specific point in time in the event of a failure — with email archiving,” the report says. “Email archiving points to the need to identify what needs to by saved, why it needs to be saved, and putting in place the technology resources necessary to archive email and be able to reproduce it in the event of an inquiry or litigation.”

The survey report states that 45.9 percent of the 1,000 organizations surveyed consider email archiving the responsibility of individual employees. Another 25.5 percent consider it part of an overall information management strategy, compared to 8.4 percent who see it as a standalone application. AIIM’s report on the survey says most organizations consider archiving a collection of massive .pst backup files.

That last point is shocking actually since there are numerous of problems with PST files in this situation like, no retention capabilities, no indexing or easy search, auditing or even security.

The rumors have been swirling for some time that Symantec had been courting Vontu, but now industry reports as well as industry sources close to the deal say a Symantec acquisition of the data leakage prevention company is now official and could be announced today.

http://www.darkreading.com/document.asp?doc_id=136241

With the increasing amount of data being submitted and stored in enterprises it is becoming more and more difficult to actually find information.  In 7 years a search on my own data for instance on Exchange would probably yield thousands and thousands of hits without giving me any particular guidance on the value of those hits or if they were business records, technical emails or other.   This post is kinda a follow up to the perimeter ‘data leakage prevention’ one I wrote about a while back about.  

Automatic Content Categorization is also a rising star in this industry and there are good reasons why.  It allows you pre-archive scan the content of emails and classify if it would fall under a specific category (spam, personal, newsletters etc).   Vendors like MessageGate and Orchestria offer solutions that allow for adding policy related data based upon the content of data.  By classifying and categorizing the data before it hits an archive it allows an organization be much more agile when searching or purging out information.  

I’m not a fan at all of manual classification. Last week Quest Software signed an agreement with Titus Labs.  Titus Labs sells software that allows for manual end user classification. End User classification is one of the frontiers that some organizations are betting on and they say that End User Classification is the way to go because the automatic systems are not accurate enough.  The idea behind this is that end users should be forced to classify information in specific business or personal related categories. 

In Exchange 2007 Microsoft released what it calls ‘managed folders’ which is a set of folders that can appear in a users mailbox to which this user can drag a message to and then tags this message with a category. While on the surface this end user classification sounds like a terrific idea, but in my opinion relying on end users to correctly classify information is fundementally flawed.  End users make mistakes, end users have better things to do to be productive.  This image I found says more then a 1000 words:

This year marks the 9^th time I was awarded the MVP award for Exchange Server from Microsoft. Each year MVP Awardees have to prove themselves over again to be ‘worthy’.  

What are MVPs? 

There are lots of benefits with being an MVP. Microsoft Most Valuable Professionals (MVPs) are exceptional technical community leaders from around the world who are awarded for voluntarily sharing their high quality, real world expertise in offline and online technical communities. Microsoft MVPs are a highly select group of experts that represents the technical community’s best and brightest, and they share a deep commitment to community and a willingness to help others.MVPs represent a broad spectrum of Microsoft product users. They occupy many different professions including accountants, teachers, artists, engineers and technologists. MVPs reside in over 90 countries, represent 30 different languages, and cover more than 90 Microsoft technologies. MVPs tend to be early adopters of new technology and actively communicate their experiences to millions of other technology users. Through their extensive community activity, MVPs help others solve problems and discover new capabilities, helping people get the maximum value from their technology.Each year, the technical community’s best and brightest are awarded as Microsoft MVPs.  Microsoft is pleased to recognize and award its Most Valuable Professionals (MVPs). We present the MVP Award to thank individuals for their exceptional contributions to technical communities worldwide. When a community participant sees an MVP in a technical community, whether in a newsgroup, as a user group host, a conference speaker, or a respondent in forums, that community participant can be confident that the information shared by the MVP will be of the highest caliber and will help every user make the most of the technology.

Microsoft also benefits from engaging with MVPs through conferences, user groups, code camps, the MVP Global Summit, and other events. MVPs share their independent, real-world feedback with us, thereby helping Microsoft better understand users’ needs, improve current products, and develop future technology.

Customers and technology users who work with Microsoft technologies are encouraged to visit user groups, conferences, and training sessions that are led by MVPs. MVPs are objective technology experts who are eager to share their knowledge. They have no obligation to Microsoft and freely share their expert opinions and experience, earning users’ respect and trust.

OK, I do have to admit that figuring out Microsofts licensing program is not for the faint of heart, but as far as licensing SQL for archiving products goes it isn’t that difficult.   While there are vendors out there that allow their archiving software to run on other databases then SQL, the defacto standard in this industry really is Microsofts SQL Server.  In all my years in this industry I have run across exactly 2 customers who did not have SQL running in their environment, but of course that is just my view on this.

All archiving products require a database, they use it for storing configuration information, metadata and sometimes more (up to storing messages and attachments).  The SQL database is truly the backbone of a proper functioning archiving product, if SQL fails .. most likely your archiving product will stop functioning as well.   Database storage is a complete other industry and I’ve had some interesting talks with the SQL folks on how large databases can scale.  While doing research on this post I ran into this article on the worlds largest database deployments and it might be worth checking these numbers out.   The largest Windows deployed SQL Server (the latest numbers I could quickly find were from 2005) is close to 20TB which is a number that probably no archiving deployment ever is going to hit.

Lets get back to licensing .. there are 4 flavors of Microsoft SQL Server 2005:

• SQL Server 2005 Express
• SQL Server 2005 Workgroup
• SQL Server 2005 Standard
• SQL Server 2005 Enterprise

SQL 2005 Express replaced the old SQL Personal Edition or MSDE and its real limitation is that the database can only grow to 4Gb max.  This makes it a viable solution to be used in a very small archiving environment, however the database limitation is definately a concern which is one of the reasons why many archiving vendors do not support running their archiving software leveraging on SQL Express.  In almost all environments customers will either leverage an already available SQL Server or will deploy a new SQL server if the need for a dedicated machine exists.

While I can’t claim what you will need in your environment as far as licensing goes, there are a few options that you have based upon the amount of users in your environment leveraging the SQL Server.  Even though end users are not accessing the SQL Server directly they do leverage the ’services’ from a SQL Server by using an archiving system.  Microsoft calls this ‘multiplexing’ and in the licensing FAQ document they state the following:

Sometimes organizations develop network scenarios that use various forms of hardware and/or software that reduce the number of devices or users that directly access or use the software on a particular server, often called multiplexing or pooling hardware or software. Use of such multiplexing or pooling hardware and/or software does not reduce the number of CALs required to access or use SQL Server software. A CAL is required for each distinct device or user that is connected to the multiplexing or pooling software or hardware front end.

The rule of thumb is that a processor license is cheaper then single CALs from about 100 users and higher.  However if you are unsure whether a CAL is required, you should contact your Microsoft sales representative or account manager. Inquires can be directed to the Microsoft Sales and Partner Information line by calling (800) 426-9400.

This is kinda off topic but still something that I wanted to post.  The other day I did some ‘garage cleaning’ and I ran into a box with some VAX VMS manuals (yes I’ve been around that long).  The sheer weight of these and amount of paper made me realize on how technology has changed things in a different way.  These days you rarely will get a binder or even a paperback with your software purchase.  Most software manuals come handily on the media in Adobe’s PDF format.

This made me think that Adobe alone has been responsible for saving billions of trees that didn’t have to get chopped up for our computer manuals.  In the same way the electronic records that now are stored on magnetic or optical platters are no longer piles of paper collecting dust in a big warehouse.

 Indeed .. software can be good for the environment.

I’ve seen some bad ways in my career on how some archived data is stored for long time storage. The worst cases where involving using Exchange Public Folders and Exchange Databases.

I simply can’t understand why you would archive data from an Exchange Server and then put it back into Exchange.  In that case why not just leave it in Exchange from the beginning ?