Post Process

Everything to do with E-discovery & ESI

Case Summary; Nat’l Day Laborer Org; 2nd Cir.; Adequacy of Searches under FOIA

Posted by rjbiii on September 18, 2012

Context: Three entities, the National Day Laborer Organizing Network; the Center for Constitutional Rights; and the Immigration Justice Clinic for the Benjamin N. Cardozo School of Law filed a lawsuit seeking compliance of targeted agencies to earlier Freedom of Information Act (FOIA) requests.  Requests were sent to: Immigration and Customs Enforcement (ICE); the Federal Bureau of Investigations (FBI); the Department of Homeland Security (DHS); and the Office of Legal Counsel (OLC).  The plaintiffs were seeking information related to “Secure Communities,” a federal program launched in 2008.  It has long been the practice for local law enforcement agencies to send fingerprints of individuals they’ve arrested to the FBI to be checked against those in the national criminal history database. 

 Under Secure Communities, those prints were also sent to DHS to be compared to immigration records.  Plaintiffs also requested information associated with the issue of whether and how local agencies could “opt-out” of the participation in Secure Communities. After the agencies conducted extensive searches involving thousands of hours and resulting in tens of thousands of records produced, Defendants and Plaintiffs cross-moved for summary judgment on the adequacy of the searches.

 Issues:  Requesting parties lodged two main criticisms of the agencies search protocols. 

 First, they contended that the agencies failed to conduct any searches over records of certain custodians who were likely to possess records. 

 Next, they argued that the producing parties failed to establish that those searches they did conduct were adequate. In elaborating on their second criticism, plaintiffs contended that many of the submitted affidavits were insufficiently detailed to permit a finding by the court of adequacy; and that where searches were described sufficiently, those searches were inadequate.

Legal Standard: A FOIA request at Summary judgment has different standards and burdens than would be the case for a litigant in a civil matter.  The burden to “prove” the adequacy of a search protocol is much heavier for the agency in this situation. That said, one must wonder in which circumstances listed below the heightened the standard makes a difference.  I’ll sprinkle commentary on this and other subjects throughout my summary of the court’s analysis below (always in italics).  A general point should be made on this, however.  FOIA cases are such that they invite analyses of search protocols and methodologies.  After all, the entire case centers around what was, or should be, produced. In a civil matter, litigants are likely to be granted greater levels of deference.  However, once the issue of e-discovery methodologies arises, the standards might not vary as greatly as one might imagine.  This is especially true if the court is presented with evidence that relevant documents were omitted from production.

 Another item to note is that the searches discussed here are associated with collection, rather than downstream processes occurring during processing or review.  Search protocols for collections deserve a heightened level of scrutiny.  If the protocol is deficient, data remains uncollected, and becomes susceptible to deletion or loss.  Narrowly tailored collection methodologies must be well designed.

 In order to win summary judgment under FOIA, an agency must show beyond material doubt that it has conducted a search reasonably calculated to uncover all relevant documents. The agency must conduct a search reasonably designed to identify and locate the requested documents, but is not expected to take extraordinary methods to find the records. At summary judgment, an agency bears the burden of showing that its search was adequate.  It may do so by submitting affidavits or declarations containing facts illustrating the sufficiency of the search.

 Summary judgment is inappropriate where the agency’s response raises serious doubts as to the completeness of the search, where the response is patently incomplete, or where the agency’s response is for some other reason unsatisfactory.  A showing of tangible evidence that defendants have not met their burden may defeat a summary judgment motion.  A showing of bad faith is not necessary to defeat a motion for summary judgment.

 Defendant’s affidavits must identify the files searched and provide a general description of the agency’s file system, and articulate reasons why further searching would be unlikely to disclose additional relevant information.  Agencies must establish that they search all custodians who were reasonably likely to possess responsive documents, while setting forth search terms and “the type of search performed.”  The methods used during the search, and not the fruits of the search, are the primary determinants of a search protocol’s validity. However, evidence that relevant records were not released may influence any such analysis.

 

Analysis: 

Decisions on Whose Records Should be Searched

Failure to search an obvious custodian’s records for a year and a half is unreasonable and inadequate as a matter of law, perhaps especially in light of the 56 documents returned by a subsequent search.  However, no sanctions were applied for this failure, due to the rigorous search that the agency eventually did conduct.  Failure to search a leader’s records for a subset of the records associated with a specified time period is not justified when the leader is known to have been involved with the issue at hand outside of the targeted time period and the agency did not specify at what point in time the leader became involved in the matter in its affidavit.

An agency fails to meet its burden in establishing an adequate search when it determines an office was unlikely to possess relevant records despite the office having at least one employee involved in discussions on relevant matters, the office playing a direct role in the matter at issue, and the office being tasked with providing updates on the program at issue in the case. It also fails to meet its burden if the office only searches two custodians’ records while other custodians were intimately involved in the program at issue. 

An agency’s response that an office was not likely to have records is not rendered unreasonable by the fact that the head of that office appeared at a single meeting discussing the program at issue. 

An agency may not simply assume that an office or organization does not have relevant records merely because that office never responded to the agency’s request to search. An agency fails to meet its burden when it does not conduct a search of an office or organization known to have been involved in the program at issue, yet does not explain the lack of a search or declare that the office is unlikely to possess relevant records.

An agency is required to re-draft an affidavit when the initial document failed to relate in sufficient detail search methodologies used to investigate records held by outside contractors. Upon reviewing the revised affidavit, requesting party then may decide whether or not to continue to press their complaints on this item.

An agency meets its burden when an office searches its records and claims in an affidavit to have no relevant records where no evidence exists to suggest otherwise. Although requesting party’s to initial conclusion was reasonable that the office might possess relevant data given the offices’ work and a statement from one employee that he had been “tracking” the matter for some time, some degree of deference is granted to the producing party because they are more familiar with their work and record systems than are the court or requesting parties.

An agency fails to meet its burden when an office known to have significant amounts of responsive documents searches 22 of its active employees’ files, but fails to search the records of seven departed employees. 

Datasets of departed employees are often overlooked.  In those environments where everything associated with projects is stored on network shares, some of the more burdensome measures may be avoided.  However, those archives that are exclusive to that custodian and might contain relevant documents, such as e-mail, should be properly maintained and available for collection.  Similar issues arise with hardware or software application retirements.

An agency meets its burden when an office searches paper and electronic correspondence of a leader whose role likely doesn’t require that leader to generate her own documents that wouldn’t be shared with others.  The search over correspondence examined a universe of documents reasonably calculated to uncover all responsive documents.

An agency has the responsibility of following up on obvious leads, such as recipients of emails that discuss issues relevant to the matter.

Analysis of Search Methodologies 

In earlier times, custodians and searchers were responsible for becoming familiar with the scope of requests and examining documents individually to determine their responsiveness. Now, custodians use computers to do the searching, and knowing the precise instructions that custodians give their computers is crucial to determining the adequacy of a search.

In the past, a court largely relied on the discretion of the searching parties to determine whether a document was responsive; but at least in that era, courts knew that the searching parties were actually looking at the documents with their eyes. With most electronic searches, custodians never actually look at the universe of documents they are searching. Instead, they rely on their search terms and the computer to produce a subset of potentially responsive records that they then examine for responsiveness.

Defendants argue they need not disclose with meticulous documentation the details of an “epic search,” and that custodians should be trusted to execute searches correctly because they use the necessary skill set in their daily work.  The court disagreed, giving two rationales.

First, custodians cannot “be trusted to run effective searches,” without providing a detailed description of those searches, because FOIA places a burden on defendants to establish that they have conducted adequate searches; FOIA permits agencies to do so by submitting affidavits that “contain reasonable specificity of detail rather than merely conclusory statements.”

Second, most custodians cannot be “trusted” to run effective searches because designing legally sufficient electronic searches in the discovery or FOIA contexts is not part of their daily responsibilities. Searching for an answer on Google (or Westlaw or Lexis) is very different from searching for all responsive documents in the FOIA or e-discovery context. Simple keyword searching is often not enough.

There is a “need for careful thought, quality control, testing, and cooperation with opposing counsel in designing search terms or ‘keywords’ to be used to produce emails or other electronically stored information.” And beyond the use of keyword search, parties can (and frequently should) rely on latent semantic indexing, statistical probability models, and machine learning tools to find responsive documents. Through iterative learning, these methods (known as “computer-assisted” or “predictive” coding) allow humans to teach computers what documents are and are not responsive to a particular FOIA or discovery request and they can significantly increase the effectiveness and efficiency of searches. In short, a review of the literature makes it abundantly clear that a court cannot simply trust the defendant agencies’ unsupported assertions that their lay custodians have designed and conducted a reasonable search.

When custodians do track and report their search protocol properly, the court’s determination of adequacy is highly context-specific.  Often, analysis of the effectiveness of criteria used to execute the search and an iterative process of criteria modification and re-execution of queries may be necessary to assure that a search is reasonably calculated to uncover all responsive material.

The involvement of custodians is instrumental to an effective collection protocol.  Custodians associated with the project or incident in question will often be able to identify the most significant sources of relevant data.  However, a search methodology which relies on custodians to execute complex technology-based searches is an inviting target for criticism.  The greater the number of custodians involved, the greater number of “searchers” exist who can be scrutinized.  Consistency may become an issue.  Did all custodians run the protocol with identical (or nearly identical) effort, attention to detail, and methods? Were all custodians sufficiently proficient with the technology used?  Was the documentation generated the same way in each collection? Finally, would any custodian have a motive to omit particular records from the collection?  A skilled, knowledgeable, and objective individual who would execute queries in consistent fashion would deflect many of the obvious criticisms that could be leveled at “self-collecting” custodians.

Plaintiff’s e-discovery expert argued that “there is no indication that [the agencies] undertook any analysis” to review the effectiveness of their criteria, and there is an “absence of any evidence of a thoughtful process in selecting and testing search terms.”  The court agreed that many of the searches weren’t “perfect.” In the absence of evidence showing the efficacy of the terms used, it is impossible to determine the adequacy the search protocol. While repeating “vast swaths” of the search to ensure adequacy is a waste of resources, and the costs to all parties would surely outweigh the benefits to plaintiffs, FOIA requires the government to respond adequately to requests.  Rather than revisit old searches, the parties must cooperate to design and execute a small number of new, targeted searches.

Here, the court continues the march towards numbers-based analyses. An iterative process that provides solid data to validate the initial search criteria, subsequent modifications, and decisions to omit certain data sources has become something of a Holy Grail.  But does a court really want to set a numerical threshold exempting data sources from further consideration? How does one calculate the proper number that triggers the determination that although some relevant documents may exist in a particular repository, the effort of collecting, processing, reviewing, and producing from an archive is not reasonable? Proportionality is a touchstone of the current process, but it is not clearly defined, and is generally tethered closely to the circumstances of individual matters.  Coming up with an “reasonableness quotient” may be beyond the abilities of the average court.

Custodians who should have searched their records must now do so.  A sample of the custodians who conducted searches but failed to provide the Court with appropriate details regarding the methodologies behind those searches must conduct new, fully-documented searches.  A smaller group of the custodians who listed the search terms but did not provide evidence on the efficacy of those terms must repeat the searches to allow the court to efficiently evaluate whether the initial searches were adequate.  The parties will need to agree on search terms and protocols, and might need to conduct testing in order to evaluate and refine those terms.

The parties may also agree to use innovative methods, such as predictive coding techniques, if desired.  The court then listed specific custodians that would be required to conduct new searches.

Here we see a court not only accepting predictive coding, but nearly recommending its use on its own volition.  In the opinion, at least, there is no reference to plaintiffs or defendants proposing “innovative methods.”  It is possible that the significant time, effort, and cost already associated with document searches—many of which must now be repeated—led the court to make the offer.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: