27 August 2007.
Several weeks ago, a legal advocacy group
issued a press release, which informed about
the organization's efforts on behalf of
teenage girls who had been abused in a
detention center. It referred readers to a
redacted document on its Web site for more information.
mother of a teenage girl, it sparked my interest. I
displayed the redacted PDF
document, and then examined its security. Since I was
able to discover the names of the girls, I informed the
group, who quickly corrected the flawed document.
But what if my motives were not
that of a curious and outraged parent?
Stories about improperly redacted documents
appear frequently in the news and legal literature. Often, those
who discover the redacted information expose it. But
the motives of researchers run the gamut from mild
curiosity to winning at all costs. Thus, while exposure
might not be desirable, use of the
information without the creator's knowledge or consent
As was the case in this example,
such findings often involve serendipity. But luck isn't
always a factor. Strategy plays a major role in certain
types of research; for
instance, competitive intelligence. It behooves
companies to learn about these techniques in order to
protect their confidential information.
Private - Keep Out!
When researchers want to know something about a company, one
of the first places they check is its Web site. They read
what the company wants them to know. Then, if they want to
dig deeper continuing to use the company itself as a
source, they check two things: the
Internet Archive and the Web site robots exclusion
The former archives previous versions of the site. As
I relate in an earlier
article, these sometimes shed light on information
the company might not want to reveal.
Because of improved security at Web sites, robots exclusion files
generally are not as helpful as they used to be. But
researchers still check them, and so should you.
The files contain commands that instruct search
engines about areas of the site they should not index.
Any legitimate search engine will obey these commands.
To work correctly, the file must appear in the root
directory of the Web site. It must bear the filename,
robots.txt. Therefore, to find it, you enter:
They are easy to read. The one on The Virtual
Chase looks, in part, like this:
The user-agent is the targeted crawler (search
engine). The asterisk is a wildcard. Each character
string following the command, disallow, is a
subdirectory. Consequently, this abbreviated set of
commands tells all search engines not to crawl the
subdirectories labeled, _private, cas,
cir and data. A researcher, of course, will
choose to attempt entry, or not.
It's like placing a Keep Out sign on a door. If the
door isn't locked, someone may walk through it.
As I explain in the above-referenced article on the
Internet Archive, a prospective client approached
a group of my firm's lawyers about launching a new
business in an industry with an unsavory reputation. One
of the conditions for considering representation was
that the woman not have prior dealings in the industry.
She claimed she did not.
Research at the client's business Web site in the
Internet Archive, however, uncovered circumstantial
evidence of several connections. Through telephone
research and public records, we were able to verify that
not only was she working in the industry, she was
the subject of a then-active federal criminal
Clues about information you would rather researchers
not discover often come from the company itself. In a
recent and widely
publicized example, Google inadvertently released
information about its finances and future product plans
in a PowerPoint presentation.
Searching for Microsoft Office files is, in fact, an
research strategy because the meta data often reveals
information the producer did not intend to share. You
may tack on a qualifier or use a search engine's
advanced search page to limit results to specific file
types, such as Word documents (doc), PowerPoint
presentations (ppt) or Excel spreadsheets (xls).
At Google, the qualifier is
filetype: whereas at Yahoo it is
Enter the file extension immediately after the colon (no
spaces). Check each search engine's help documentation
for the appropriate qualifier, or consult a Web site,
Search Engine Showdown, which tracks and informs about
Searching certain phrases
sometimes produces intriguing results. Try the phrases
individually to discover the potential for this technique
when coupled with a company, organization or agency name:
You might find additional ideas
for searching dirty in the
Google Hacking Database.
Book search engines, such as
Google Book Search and the
Text Archive at the Internet Archive, are
becoming increasingly valuable in research. If you
uncover even a snippet of relevant information online,
it may save you valuable research time offline.
One of my recent success stories involves finding an
entire chapter on the target company in a book published
just a few months prior to the research. Of course, I
was unable to read the book online. I had to purchase
it. But the tools helped me find what I might have
missed without them.
this is not the underground to which I refer. By using
these tools, you are not skirting the process for
rewarding those who wrote and published the book.
The underground, while eminently real, is not so much a
place as it is a mindset - one that sets information
free. The result is a mixed bag of commercial products,
including books, music, digital artwork, movies and
software, that have been copied or reverse engineered.
Try the search strategy below.
Replace the phrase, Harry Potter, with the keywords
of your choice:
"index of" "last modified
size description" "parent directory" "harry potter"
of the search statement preceding "harry potter"
comprises a strategy for finding vulnerable Web sites or
servers. In a
nutshell, it commands the search engine to return
matches to directory structures instead of single Web
pages. If a Web site is properly secured, the search
engine will be unable to provide this information.
To some extent, you can monitor the availability of
files that comprise unauthorized copies of products by setting up what Tara Calishain calls
information traps. Tara's excellent book on
Information Trapping provides many examples of ways
to monitor new information.
One possibility is to use the above search strategy for
best-selling or popular products, and then set up a
Google Alert for new matches to each query.
While you should monitor hits at other search engines
besides Google, doing so requires more work. First, test
and perfect the query so that you are retrieving useful
results. Set the search engine preferences to retrieve
100 items per page. Then copy the URL when the search
results display. Paste it into a page monitor, such as
The tracking software or Web service will monitor
changes in the first 100 search results. You may opt to
have it send the changes to you by e-mail.
Companies and other organizations that want to protect
proprietary or confidential information should conduct
this type of research with regularity. You can expedite
some of the search process with information traps. But considering
the stakes, regular thorough searching is a worthwhile investment.