8 March 2005. Strategic and competitive intelligence (CI) research used to be conducted through searches of the fee-based online services, interviews with current and former employees of a company, skulking around the target company's exhibit booth at an industry conference, and digging through public records. Interestingly, a lot of strategic information can now be gleaned from a simple Web search -- no dumpster-diving required. I'll preface the rest of this article by noting that it is up to you and your organization to decide what your tolerance is for this type of research.

It is wise to assume that anything you or your staff do may be written up on the front page of The New York Times.

Try typing "company name" AND "company confidential" (or AND "internal use") in any of the major search engines. Wow. You'll get similarly interesting results if you search for "price list" or "salary" along with the company name.

So, what do you do if you find this information? You’re an info pro, so presumably your client (whether internal or external) expects you to conduct as thorough a search as possible. Do you pass along this information on the assumption that, if it was spidered by a search engine, it should be considered public information? Alternatively, do you contact the company and tell them that their Web site security is getting a bit leaky? Do you contact your organization’s general counsel and ask for help?

I don’t know the answers to these questions. But as some competitive intelligence professionals have learned to their chagrin, it is wise to assume that anything you or your staff do may be written up on the front page of The New York Times. What makes this issue so intriguing is that more information is available on the Web, and we info pros are becoming more adept at finding hidden or deep Web content.

The old formula of being able to maintain privacy by obscurity -- "What are the odds my employer will find that edgy newsgroup on hallucinogens that I participate in?" -- no longer applies. How "deep" should you go when researching an individual? Are postings in a non-work-related newsgroup or e-mail discussion list fair game? What about the personal information they post in a social networking service such as Linked in or orkut?

Here’s another deep Web research quandary. Say I discover one page from a competitor, labeled "confidential," and I work backward through its URL to a section of the company’s Web site, which contains other material that probably is not intended for public dissemination, but is in fact accessible. Is that ethically equivalent to someone walking into my house if I leave the front door open? Or is anything the company puts on its publicly-accessible Web site considered public?

Fortunately, there are plenty of resources from the CI community on what is and isn’t considered ethical behavior. See, for example, the Society of Competitive Intelligence Professionals’ resources. Unfortunately, much of this information was written pre-Web, or at least prior to the explosion of information available on the Web. Your organization may have more stringent rules regarding what is and isn’t considered public information. If you work for a multinational organization, you may also be bound by the far more restrictive rules of European countries regarding personal information.

In any event, give this some thought. Find out what others within your organization think about this type of search. What's your comfort level?

© 2004, 2005 Mary Ellen Bates all rights reserved.

Mary Ellen Bates is the principal of Bates Information Services, a research and consulting business based in Boulder, CO.