Case Blurb: McBride, Distinction made between whether to execute a search and “ability to find” data

[Requesting Party] claims that the defendants violated a government-imposed document preservation order if they are not able to produce [certain] e-mails. It is important to keep separate, and not conflate, two distinct concepts: whether an additional search for certain information should be conducted, and whether or not the inability to find certain information should be sanctioned. The first is a question of interpreting the Federal Rules of Civil Procedure, while the second is a matter of the application of the common law principle of spoliation.

U.S. ex rel McBride v. Halliburton Co., 05-CV-828 (D.D.C Jan. 24, 2011)

Read a case summary here, from Bow Tie Law.

Court tells Artist to Come Clean

A federal court has ordered Shepard Fairey, the man behind the now famous “Hope” poster of Barack Obama to disclose ESI that was deleted, and to reveal the identity of the person behind the destruction of evidence. From the ABA’s story on the matter:

Fairey switched lawyers in the case last November after his then-counsel said in a court document that the artist had deleted electronic files to cover up a mistake about which photo he had used. The document, a motion for new counsel, said the events presented a potential for conflict of interest between Fairey and his then-lawyers.

Fairey is under criminal investigation in connection with the disclosure, according to court filings.

On Monday, [Judge] Hellerstein ordered Fairey and his companies to disclose how computer files were searched for relevant legal records, according to the AP story.

Case Summary: William A. Gross Constr. Assocs.; Court Issues ‘wake up’ call to counsel on keywords

William A. Gross Constr. Assocs. v. Am. Mfrs. Mut. Ins. Co., 2009 U.S. Dist. LEXIS 22903 (S.D.N.Y. Mar. 19, 2009)

The opinion addressed the creation of a search protocol in this case over “alleged defects and delay in the construction ofthe Bronx County Hall of Justice.” The court began by lecturing counsel, and the district’s bar in general, over its (evidently perceived lax) attitude toward formulating search criteria:

This Opinion should serve as a wake-up call to the Bar in this District about the need for careful thought, quality control, testing, and cooperation with opposing counsel in designing search terms or “keywords” to be used to produce emails or other electronically stored information (“ESI”). While this message has appeared in several cases from outside this Circuit, it appears that the message has not reached many members of our Bar.

The lesson began, the court complained that counsel had not adequately communicated in order to arrive at a suitably agreed-upon search criterion, and the protocol that had been formulated was not the result of a sufficiently methodological process:

This case is just the latest example of lawyers designing keyword searches in the dark, by the seat of the pants, without adequate (indeed, here, apparently without any) discussion with those who wrote the emails.

The court then stated that it found itself in the “uncomfortable position of having to craft a keyword search methodology for the parties, without adequate information…”

After making certain additions to the keywords comprising the search criterion, the court then cited opinions from Victor Stanley, Inc. v. Creative Pipe, Inc., 250 F.R.D. 251, 260, 262 (D. Md. May 29, 2008) (Grimm, MJ.) and United States v. O’Keefe, 537 F. Supp. 2d 14, 24 (D.D.C. 2008) (Facciola, MJ.) for supporting the supposition that formulating keywords “requires careful advance planning by persons qualified to design effective search methodology” and designing search protocols “is a complicated question involving the interplay, at least, of the sciences of computer technology, statistics and linguistics.” The court continued by noting that the “best solution” was for opposing counsel to cooperate in formulating a search protocol, and strongly endorsing the Sedona Conference’s Cooperation proclamation.

Lessons learned:

1. The formulation of search protocols is to be taken seriously, and accomplished with careful thought, quality control, and testing.
2. Initial assumptions should not be generated blindly or in an arbitrary fashion, and should be tested and, if necessary, modified.
3. Counsel should be prepared for greater levels of communication, cooperation, and transparency than in the past.
4. Even an admittedly imperfect protocol, if the above elements are present, may suffice.

Case Blurb: William A. Gross. Constr. Assocs., Inc; Court helps to craft Search Protocol

This case involves a multi-million dollar dispute over alleged defects and delay in the construction ofthe Bronx County Hall of Justice, also known as the Bronx Criminal Court Complex. The Dormitory Authority of the State of New York (“DASNY”) was the “owner” of the project. Non-party Hill International is DASNY’s current construction manager, and DASNY has agreed to produce Hill’s project-related documents and ESI to the other parties to this suit. The issue before the Court is the production of Hill’s emails, stored on its server at its New Jersey office, and how to separate project-related emails from Hill’s unrelated emails.

DASNY’s proposed search terms included “DASNY,” “Dormitory Authority,” and “Authority,” FN1 and the names of the other parties to the action. DASNY also added “Court! in connection with Bronx,” “Hall of Justice,” and “Bronx but not Zoo” (since Hill worked on an unrelated project involving the Bronx Zoo). The other parties requested the use of thousands of additional search terms, emphasizing the construction issues they were involved in, such as “sidewalk,” “change order,” “driveway,” “access,” “alarm,” “budget,” “build,” “claim,” “delay,” “elevator,” “electrical” — you get the picture. DASNY correctly pointed out that use of such extensive keywords would require production of the entire Hill email database, since Hill’s business is construction management, and those terms would be used for any construction project.

FN1: The Court is no keyword expert, but if one is searching for “Authority,” to also search for “Dormitory Authority” is clearly redundant.

Hill’s only contribution to the discussion was to agree that DASNY’s search [*10] terms were probably too narrow but the other parties’ terms were overbroad, and that Hill did not want to produce emails that did not relate to the Bronx Courthouse project. This problem would have been avoided, of course, if Hill used a standard “Re” line in its Bronx Courthouse emails to distinguish that project from its other work. It did not do so, however. Moreover, while Hill was in the best position to explain to the parties and the Court what nomenclature its employees used in emails, Hill did not do so — perhaps because, as a non-party, it wanted to have as little involvement in the case as possible.

That left the Court in the uncomfortable position of having to craft a keyword search methodology for the parties, without adequate information from the parties (and Hill). The Court ruled at yesterday’s conference that in addition to DASNY’s proposed terms (including variations on and abbreviations of party names), the search should also include the names ofthe parties’ personnel involved in the Bronx Courthouse construction. FN2

FN2: The Court acknowledges that this result is less than perfect, and that there is a risk that as information later comes out at depositions of the Hill employees, another search may have to be done.

William A. Gross Constr. Assocs. v. Am. Mfrs. Mut. Ins. Co., 2009 U.S. Dist. LEXIS 22903 (S.D.N.Y. Mar. 19, 2009)

Case Blurb: William A. Gross. Constr. Assocs., Inc; Need for cooperation and communication during E-Discovery

Electronic discovery requires cooperation between opposing counsel and transparency in all aspects of preservation and production of ESI. Moreover, where counsel are using keyword searches for retrieval of ESI, they at a minimum must carefully craft the appropriate keywords, with input from the ESI’s custodians as to the words and abbreviations they use, and the proposed methodology must be quality control tested to assure accuracy in retrieval and elimination of “false positives.” It is time that the Bar — even those lawyers who did not come of age in the computer era — understand this.

William A. Gross Constr. Assocs. v. Am. Mfrs. Mut. Ins. Co., 2009 U.S. Dist. LEXIS 22903 at *13-14 (S.D.N.Y. Mar. 19, 2009)

Case Summary: Rhoads Indus.; Attorney Client Privilege, the Waiver of Privilege, and Reasonable Precautions

Case Summary: Rhoads Indus. v. Bldg. Materials Corp. of Am., 2008 U.S. Dist. LEXIS 93333 (E.D. Pa. Nov. 14, 2008 )

In a question of the adequacy of a keyword screening protocol, the court entertains a motion to deem plaintiff had waived privilege as to approximately 800 inadvertently produced documents. Rhoads filed the action against Bldg. Materials Corp. of America (“GAF”) for, inter alia, breach of contract, and against R.W. Cooper & Co. (“Cooper”) for negligent misrepresentation.

Rhoads began preparing for the litigation in February, 2007. In June 2007, upon realizing that e-discovery would be extensive in the case, he hired Salvatore Gramaglia as an IT expert. Mr. Gramaglia, after conducting tests, Mr. Gramaglia purchased a license for “Discovery Attender” to be used in electronic searches. Mr. Gramaglia and Kimberly Buchinsky, an associate at Gowa Lincoln P.C., began identifying mailboxes containing relevant information to Rhoads’ project with GAF. The court stated that “Gramaglia and Buchinsky reasonably believed that the computer program would screen out all privileged materials.” Efforts to settle the dispute were unsuccessful, and on November 13, 2008, Rhoads filed the complaint. Discovery commenced shortly thereafter. Sam Washawer, a was bought in by Rhoads for the litigation. He discussed the scope of e-discovery with Mr. Gramaglia and Ms. Buchinsky, and observed the performance of Discovery Attender. He also conferred with Gramaglia and Buchinsky over the topic of search terms for searches.

During January and February of 2008, Gramaglia performed searches using terms supplied by counsel, initially identifying 210,635 documents as “being responsive.” His priv filter consisted of searching the “address line of all e-mails for these terms: *rhoadsinc* and either *gowa*, *ballard*, or *cpmi*. “Gowa” and “Ballard” represents Rhoads’ law firms, while “CPMI” refers to a non-testifying expert for plaintiff. This search resulted in the designation of 2,000 documents as “privileged,” although they were not placed on a privilege log at that time.

In response to the large volume of documents initially returned, counsel narrowed the responsive search criteria, reducing the total documents returned to 78,000 that plaintiff believed were responsive and non-privileged. Ms. Buchinsky then manually reviewed documents from specific mailboxes for privilege. She removed certain documents from the production, and entered them on a privilege log. Meanwhile, counsel also reviewed twenty-two boxes of hard copy documents for responsiveness and privilege.

On May 13, 2008, Rhoads produced to Δ’s, three hard drives containing responsive electronic documents, including the 78,0000 documents returned by the responsive “screen.”

On June 6, 2008, Rhoads produced two privilege logs to Δ’s, in response to a court order. One log was associated with Ms. Buchinsky’s manual review, while the other contained hard copy documents.

On June 5, 2008, Δ’s GAF notified Π’s counsel that some privileged e-mails may have been produced. Π immediately responded by asserting that no privilege had been waived, and that this was likely a case of inadvertent production. Over the next two and a half weeks, Π then conducted nine depositions and responded to Δ Cooper’s Motion to Dismiss before addressing the privilege issue.

On or about June 23, 2008, Ms. Buchinsky began reviewing the 78,000 e-mails produced as a result of the responsive searches executed in Discovery Attender. She generated a new privilege log identifying 812 emails as privileged, producing it to Δ’s on June 30, 2008 as an attachment to a letter invoking FRCP 26(b)(5)(B) to have Δ’s sequester the inadvertently produced documents.

On August 19, 2008, Δ Cooper filed a Motion to Deem that Plaintiff had waived the Privilege to the approximately 800 documents. Δ GAF filed a motion joining Cooper’s motion on August 25, 2008. Defendants did not dispute that the production was inadvertent and that the software purchased by Rhoads was designed to ferret out privileged documents. Defendants did argue that Rhoads’s technical consultant and counsel were not sufficiently careful to review the software screening and to take steps to prevent disclosure when it appeared obvious that privileged material had filtered through the screening procedure.

On November 5, 2008, the first hearing on the waiver of privilege was conducted. All three privileged logs were produced. Mr. Gramaglia and Ms. Buchinsky testified as to the method of creation of the logs, and their relationship to each other. Rhoads admitted that 2,000 e-mails previously designated as privilege had not yet been disclosed in a privilege log. Ms. Buchinsky testified to her belief that a manual review of all 78,000 emails would have resulted in the timely identification of those documents in a privilege log. The court then ordered that any documents not on a privilege log as of the date of that hearing should be produced, absent exceptional circumstances.

After the hearing, Rhoads inspected the 2,000 documents earlier set aside, but not identified in a log, as privilege. The results of this inspection were discussed on a hearing on November 13, 2008. Rhoads concluded that 941 documents were duplicative. Of the 1059 unique documents, 511 were responsive, consisting of 335 privileged documents and 176 non-privileged documents. Of the 335 privileged documents, 215 had been previously identified in other logs. Rhoads agreed to produce the 176 responsive non-privileged documents, and generated a fourth privilege log. The court expressed its opinion that Rhoads might still retain privileged and un-logged documents.

The court then weighed the facts using the five factor test laid out in Fidelity & Deposit Co. of Md. v. McCulloch, 168 F.R.D. 516 (E.D. Pa. 1996). These factors are:

(1) The reasonableness of the precautions taken to prevent inadvertent disclosure in view of the extent of the document production.
(2) The number of inadvertent disclosures.
(3) The extent of the disclosure.
(4) Any delay and measures taken to rectify the disclosure.
(5) Whether the overriding interests of justice would or would not be served by relieving the party of its errors.

On the first factor (reasonableness taken to prevent inadvertent disclosure), the court found that the following factors favored the plaintiff (producing party):

1. Plaintiff purchased a special software program, Discovery Attender, for the purposes of complying with discovery in this litigation.
2. Plaintiff’s technical consultant (Mr. Gramaglia) conducted trial searches prior to purchasing the software and was satisfied with its reliability and accuracy.
3. Mr. Gramaglia was experienced with Rhoads’s computer system.
4. Plaintiff believed that its search terms would pick up all attorney-client communication.
5. Plaintiff did not use “Warshawer” or “Costello” (outside counsel) as search terms because counsel believed Gowa was included on all e-mails going to the client.
6. Plaintiff did not include “privileged” or “confidential” as keyword search terms because all Rhoads e-mails use these words in the signature line at the bottom of every e-mail, and thus use of these words would have identified every Rhoads document as privileged.
7. Ms. Buchinsky spent over 40 hours reviewing documents for privilege before production. Additional attorneys spent significant hours on the privilege review.

The court found the following factors favored defendants (requesting parties):

1. Plaintiff should have used additional search terms to weed out potentially privileged documents, especially the names of all of its attorneys. Here the court cited Bensel v. Air Line Pilot Ass’n, 248 F.R.D. 177, 180 (D.N.J. 2008).
2. Ms. Buchinsky had no prior experience doing a privilege review and her supervising attorneys did not provide any detailed oversight.
3. Plaintiff’s search for privileged documents was limited to e-mail address lines (as opposed to the e-mail body). Therefore any potentially privileged e-mails (as defined by Plaintiff’s search terms) that were subsequently forwarded outside of Rhoads and either Gowa, Ballard, or CPMI would not be captured by its search.
4. Plaintiff produced documents that its limited search should have caught. Therefore Plaintiff not only failed to craft the right searches, but the searches it ran failed. Plaintiff has no explanation for this.
5. As Stanley notes, relying exclusively on a keyword search for the purpose of conducting a privilege review is risky, and proper quality assurance testing is a factor in whether precautions were reasonable. Victor Stanley, 250 F.R.D. at 257, 260. Here there was no testing of the reliability or comprehensiveness of the keyword search.
6. Plaintiff’s only testing of its search was to run the same search again

On the second factor, the number of inadvertent documents produced, the court found this fact favored plaintiffs:

1. The number of documents inadvertently produced represent a small percentage (1-2%) of the total production.

The court found these facts weighed in favor of defendants:

1. 800 inadvertently produced documents plus at least 120 privileged documents that this Court now orders be produced is still a large number of documents regardless of the percentage produced.
2. The court in Stanley found that 165 documents was a sufficiently large number to favor waiver of the privilege.

The court stated that nothing in the record indicated whether the third factor (the extent of the disclosure) weighed in favor of one party over the other.

On the fourth factor (Any delay and measures taken to rectify the disclosure), the court found these facts favored plaintiffs:

1. Plaintiff immediately responded to Defendant’s e-mail that some potentially privileged documents had been produced by stating that production was inadvertent and inquiring which documents Defendant was referring to;
2. As relied on in Fidelity, Plaintiff was under a tight discovery schedule at the time it was notified of the inadvertent production. Fidelity & Deposit Co., 168 F.R.D. at 522. Ms. Buchinsky was helping prepare for nine depositions taken over the course of twelve days. Plaintiffs also had to respond to Defendant Cooper’s Motion to Dismiss during this time.
3. Plaintiff invoked the remedial measure of FRCP 26(b)(5)(B) to have the inadvertently produced documents sequestered at the same time it produced the June 30th privilege log (Ex. D-1), three weeks after learning of the disclosure.
4. Plaintiff was willing to produce a cleansed hard drive (i.e. without the privileged documents) to Defendants in September, but Defendants refused the hard drive.
5. Defendant did not file their Motion to Deem that the Privilege was Waived until two months after Plaintiff produced the June 30th privilege log.

Those facts favoring defendants:

1. Plaintiff had abundant time to review its own documents and segregate any privileged documents before it started the litigation and also before it began producing documents in May 2008;
2. The crunch and time pressure that Plaintiff claims it suffered in June 2008 were caused by Plaintiff not providing adequate resources (e.g. attorneys or paralegals) to the privileged communication issue;
3. Defendants had to bring Plaintiff’s error to its attention instead of Plaintiff catching its own mistake (cites Victor Stanley).
4. It took Plaintiff over three weeks to produce a privilege log of the inadvertently produced documents once it was aware of its mistake.
5. The June 30th privilege log required Defendants to cross-reference each document produced to them against the log to check whether documents were in fact privileged. Plaintiff did not offer Defendants a cleansed hard drive until September.
6. Plaintiffs did not offer suggestions to rectify the inadvertent production until October, after many depositions had been taken.

On the final factor (Whether the overriding interests of justice would or would not be served by relieving the party of its errors), the court found these facts weighed in favor of the plaintiffs:

1. Plaintiff has shown general compliance with the three conditions of Rule 502, although considering the factors in the prior case law, Plaintiff is responsible for the confusion and delay noted above;
2. The loss of privilege would be highly prejudicial to the Plaintiff;
3. Defendants have not demonstrated substantial unfairness that they have suffered because of their inability to review the privileged documents beyond having to cross-reference documents against the June 30th privilege log.

Favoring defendants:

1. Plaintiff should have conducted a more rigorous privilege review and allowing Plaintiff to retain the privilege in these documents may not deter similar conduct in the future.

The court then concluded its opinion by ruling the following:
Those documents that were never logged after designation (part of the 2000 e-mails initially designated as privileged, but not listed in a privilege log until after the Nov. 5 hearing) were not subject to protection, as a failure to identify privilege documents in a log in a timely manner violates FRCP 26(b)(5).

With respect to the documents that were inadvertently produced, the court stated that neither Victor Stanley or Amersham were controlling. The court stated that although the facts in Victor Stanley were similar to those in the instant matter, Judge Grimm’s analysis reflects, to a more significant degree than [the court] believe[s] appropriate, application of hindsight, which should not carry much weight, if any, because no matter what methods an attorney employed, an after-the-fact critique can always conclude that a better job could have been done. The court found that factors one through four favored defendants, while the fifth strongly supported plaintiffs. As defendants have the burden of proof as the moving party, the court concluded that this burden had not been met, and privilege was not waived for these documents.

Case Blurb: Rhoads Indus.; Keywords and Emails

The advent of electronically stored information (“ESI”) has been widely discussed and has already resulted in amendments to the discovery rules. Keyword and name searches are frequently employed as an initial method to screen and sort documents, but they are not foolproof. For instance, privileged communications frequently creep into e-mail “chains,” but may appear only to some participants in the chain depending on a user’s application of the “reply” and “forward” commands. These issues [invariably] led to at least some of Rhoads’ inadvertent disclosures.

Rhoads Indus. v. Bldg. Materials Corp. of Am., 2008 U.S. Dist. LEXIS 93333 (E.D. Pa. Nov. 14, 2008 )

Mirror, Mirror, on the wall…

Wired has a set of twin articles out addressing life in the age of the Petabyte. A Petabyte is 1024 Terabytes, which is 1024 Gigabytes.

The internet came into being as a tool to enhance communication and collaboration. And it has. But it has also changed the behavior of its users, and many of those actions are now logged and stored. Combine that with a growing array of tools that record and store representations of human activity (CCTV, PDA’s etc…) and you can see that more than ever, there is a growing mass of data logging the details of individual, group, and global behavior. In the vernacular of copyright, records of our actions are now, more than ever, “fixed in a tangible medium,” and available for all sorts of purposes that wouldn’t have been possible in even the recent past.

Wired’s article on the “Petabyte Age” has a chart illustrating the differences in size between a Terabyte, with various data points in between. To cut to the chase, a Terabyte is viewed as a $200 hard drive that holds 260,000 songs, while a Petabyte is the total information “processed by Googles servers every 72 minutes.” If you have a moment, click on the above link…there are some interesting data points noted.

While the first article is interesting, the second article, The End of Theory, brings home the cogent point. The article proposes that the availability of statistics based on real behavior, rather than on imperfect models, will transform science:

Sixty years ago, digital computers made information readable. Twenty years ago, the Internet made it reachable. Ten years ago, the first search engine crawlers made it a single database. Now Google and like-minded companies are sifting through the most measured age in history, treating this massive corpus as a laboratory of the human condition. They are the children of the Petabyte Age.

In the words of the article, More is not just more, “more is different.” Google research director, Peter Norvig, is quoted as saying that “All models are wrong, and increasingly you can succeed without them.” The world is becoming a great big database.

Or perhaps just a series of smaller databases. The question we face in e-discovery concerns the rapid identification, organization, and cataloging of disparate types of data. In light of discussion over the effectiveness of search terms by Judge Grimm in Victor Stanley, Inc. v. Creative Pipe, Inc., 2008 WL 2221841 (D.Md. May 29, 2008 ), it seems certain that judicial scrutiny of search criteria formulation, and the objections to that formulation from the opposition will only increase.

Wired describes our brave new word as one in which:

[] massive amounts of data and applied mathematics replace every other tool that might be brought to bear. Out with every theory of human behavior, from linguistics to sociology. Forget taxonomy, ontology, and psychology. Who knows why people do what they do? The point is they do it, and we can track and measure it with unprecedented fidelity. With enough data, the numbers speak for themselves.

The use of these statistics in the legal arena will be something to follow closely. Could they be used to measure “community standards,” for example? Lawrence Walters, a defense attorney in a Florida obscenity case, argues that they can be used to help clarify what in the past was purely subjective. A New York Times piece has the details:

In a novel approach, the defense in an obscenity trial in Florida plans to use publicly accessible Google search data to try to persuade jurors that their neighbors have broader interests than they might have thought.

In the trial of a pornographic Web site operator, the defense plans to show that residents of Pensacola are more likely to use Google to search for terms like “orgy” than for “apple pie” or “watermelon.” The publicly accessible data is vague in that it does not specify how many people are searching for the terms, just their relative popularity over time. But the defense lawyer, Lawrence Walters, is arguing that the evidence is sufficient to demonstrate that interest in the sexual subjects exceeds that of more mainstream topics — and that by extension, the sexual material distributed by his client is not outside the norm.

In the movie, The Neverending Story, the hero (named Atreyu) is forced to view his reflection in a mirror that reveals to him “who he really is,” stripped of all flattering notions. The Age of the Petabyte gives us a mirror, of a sort, to look into and reveal things about ourselves that we might have otherwise disputed. Mr. Walters is trying to use that mirror, and hold it up for the Florida jurors. Whether his mirror is sufficiently objective is a discussion for another time, but we will see these methods used more frequently over time. Atreyu handled it. It will be interesting to see what we do with it.

Case Blurb: O’Keefe; Expert Testimony needed to dispute search methodolgy in court

Whether search terms or “keywords” will yield the information sought is a complicated question involving the interplay, at least, of the sciences of computer technology, statistics and linguistics. See George L. Paul & Jason R. Baron, Information Inflation: Can the Legal System Adapt?, 13 RICH. J.L. & TECH. 10 (2007). Indeed, a special project team of the Working Group on Electronic Discovery of the Sedona Conference is studying that subject and their work indicates how difficult this question is. See The Sedona Conference, Best Practices Commentary on the Use of Search and Information Retrieval, 8 THE SEDONA CONF. J. 189 (2008), available at http://www.thesedonaconference.org/content/miscFiles/Best_Practices_Retrieval_Methods_revised_cover_and_preface.pdf. Given this complexity, for lawyers and judges to dare opine that a certain search term or terms would be more likely to produce information than the terms that were used is truly to go where angels fear to tread. This topic is clearly beyond the ken of a layman and requires that any such conclusion be based on evidence that, for example, meets the criteria of Rule 702 of the Federal Rules of Evidence. Accordingly, if defendants are going to contend that the search terms used by the government were insufficient, they will have to specifically so contend in a motion to compel and their contention must be based on evidence that meets the requirements of Rule 702 of the Federal Rules of Evidence.

United States v. O’Keefe, 537 F. Supp. 2d 14, 23-4 (D.D.C. 2008 )

Case Blurb: Creative Pipe; Not all keyword searches are created equal

While it is known that [Producing Party] and [Producing Party's attorneys] selected the keywords, nothing is known from the affidavits provided to the court regarding their qualifications for designing a search and information retrieval strategy that could be expected to produce an effective and reliable privilege review. As will be discussed, while it is universally acknowledged that keyword searches are useful tools for search and retrieval of ESI, all keyword searches are not created equal; and there is a growing body of literature that highlights the risks associated with conducting an unreliable or inadequate keyword search or relying exclusively on such searches for privilege review. Additionally, the Defendants do not assert that any sampling was done of the text searchable ESI files that were determined not to contain privileged information on the basis of the keyword search to see if the search results were reliable. Common sense suggests that even a properly designed and executed keyword search may prove to be over-inclusive or under-inclusive, resulting in the identification of documents as privileged which are not, and non-privileged which, in fact, are. The only prudent way to test the reliability of the keyword search is to perform some appropriate sampling of the documents determined to be privileged and those determined not to be in order to arrive at a comfort level that the categories are neither over-inclusive nor under-inclusive resulting in the identification of documents as privileged which are not, and non-privileged which, in fact, are. The only prudent way to test the reliability of the keyword search is to perform some appropriate sampling of the documents determined to be privileged and those determined not to be in order to arrive at a comfort level that the categories are neither over-inclusive nor under-inclusive.

Victor Stanley, Inc. v. Creative Pipe, Inc., 2008 WL 2221841 (D.Md. May 29, 2008 )