Hard Copy Documents and External Feedback

Let’s start with traditional, stand-alone hard copy documents before discussing electronic documents, including web-based publications. Your business or organization may communicate in written forms across time zones and languages via electronic communication, but some documents are still produced on paper. Offline technologies like copy machines and printers are still tools you’ll use in the normal course of business.

Letters are a common way of introducing information to clients and customers, and you may be tasked to produce a document that’s printed and distributed via “snail mail”—the traditional postal service. Snail mail is a term reflecting the time delay associated with physical production, packaging, and delivery of a document. Legal documents are still largely in hard copy print form. So too are documents addressing the needs of customers and clients who don’t—or prefer not to—access information electronically.

Age is one characteristic of an audience that may be tempting to focus on when considering who may need to receive a hard copy letter, but you may be surprised. In a 2009 study of U.S. Internet use, the Pew Research Center found that between 2008 and 2009, broadband Internet use by senior citizens increased from 20 percent to 30 percent, and broadband use by baby boomers (born 1945–1963) increased from 50 percent to 61 percent.^[4]

Socioeconomic status is a better characteristic to focus on when considering hard copy documents. Lack of access to a computer and the Internet is a reality for most of the world’s population. It’s often stated that half the world’s population will never make a phone call in their lifetime, and even though references for the claim are widespread and diverse, the idea that some people lack access to a phone is striking for many Westerners. While cell phones increasingly allow poor and rural populations to skip investment in landline networks, and wireless Internet is a leapfrog technology that changes everything, cell phones and computers are still prohibitively expensive for many.

Let’s say you work for a major bank on the West Coast of the United States. You’ve been assigned to write a letter offering a refinance option to a select, previously screened audience composed of individuals who share several common characteristics: high-wage earners with exceptional credit scores. How will you best get their attention? If you sent an email, it might get deleted as spam—an unwanted message that often lacks credibility and may even be dangerous. The audience is small, and you have a budget for hard copy production that includes mailing costs. If the potential customer receives the letter from your department delivered by an overnight courier like FedEx, they may be more likely to receive your message.

In 2005, Wells Fargo Bank did exactly that. They mailed a letter of introduction outlining a refinance opportunity at no cost to the consumer, targeting a group identified as high profit and low risk. The channels selected—print-based documents on letterhead with a mode of delivery sure to get attention—were designed to prompt a response. The letter introduced the program, highlighted the features, and discussed why these customers were among a select group to whom this offer was being extended.^[5]

In the letter, the bank specifically solicited a customer response—a form of feedback—via email and/or phone to establish dialogue. You could measure feedback in terms of response rate; in terms of data verification on income, outstanding debt, and current home appraisal values; and in terms of channel and how customers chose to respond. All these forms of feedback have value to the author.

Hard copy documents can be challenging when it comes to feedback, but that doesn’t mean involving them in the feedback process is impossible. Even in the late 1990s, most business documents were print-based. From sales reports to product development reports, they were printed, copied, bound, and distributed—all at considerable cost.

If one purpose of your letter is to persuade the client or customer to reply by email or phone, one way to assess feedback is the response rate—the number of replies in relation to the number of letters sent. Suppose your report on a new product is prepared for internal use and targets a specific division within your company. In that case, their questions about the document may serve as feedback. If your memo produces more questions than the one it was intended to address in terms of policy, the negative feedback may highlight the need for revision. In each case, hard copy documents are often assessed through oral and written feedback.

External Feedback in a Virtual Environment

Rather than focus on dust collecting on documents once produced, perhaps read, and sometimes forgotten, let’s examine document feedback from the interactive world that gathers no dust. One challenge when the Web was young involved accurately assessing audience. Why is that relevant to a business writer? Because you produce content for a specific audience with a specific purpose, and the degree to which it succeeds relates to its value. Imagine producing a pilot television program with all the best characters, excellent dialogue, and big-name stars—only to see the pilot flop. If you had all the right elements, how could it fail? It failed to attract an audience. Television often uses ratings—measurements of estimated viewer numbers—to measure success. Nielsen is the leading market research company associated with television ratings and online content. Programs that get past a pilot or first season do so because they have good ratings and are ranked above competing programs. All programs compete within a time slot or across a genre. Those that are highly ranked—those that reach the largest number of viewers—can command higher budgets and often receive more advertising dollars. Those that reach few people are often canceled and replaced with other programs that have great characters, solid writing, and hopeful stars as the cycle continues.

Business writers experience a similar process of competition, ratings, feedback, and renewal within online publishing. You want your content to be read. Just as companies developed ways to measure television program viewers—leading to rankings that influenced which programs survived and which were canceled—the Web has a system for keeping track of what gets read and by whom. You may have heard the term “page views” or “hits” to measure how many times a web page is seen. While a high number of page views can be a good thing, they don’t always reflect true engagement. A more effective measure of a document’s success is “unique visits,” which counts how many individual people visit your site, or “conversion rate,” which measures how many visitors take a desired action, like signing up for a newsletter or making a purchase.

First, let’s examine what a hit is. When a browser receives a file from a web server, it counts as a hit. Your document may be kept on your company’s web server—a computer dedicated to serving online requests for information via the Internet. The web server receives a request from the user and sends the files associated with the page; every web page contains several files, including graphics, images, and text. Each file request and receipt between server and browser counts as a hit, regardless of how many files each page contains. So if you created an online sales catalog with twenty images per page, twenty boxed text descriptions, and all the files for indicating color, size, and quantity, your document could have quite a few hits with just one page request and only one viewer.

Does a large number of hits on your document mean it was successful? Not necessarily. Hits or page views have largely been discredited as a reliable measure of a document’s effectiveness, popularity, or audience size. In fact, “hits” is sometimes humorously called an acronym for how idiots track success.

Page views count how many times a web page is viewed, regardless of the number of files it contains. Each time a user or reader views the page, it counts as one page view.

Nielsen Online and Source.com are two companies that provide web traffic rating services, and Google has also developed services to help advertisers target specific audiences. They commonly track the number of unique visits a reader makes to a website and use cookies—small, time-encoded files that identify specific users—to generate data.

Another way to see whether a document has been read online is to present part of the article with a “reveal full article” button after a couple of paragraphs. If someone wants to read the entire article, they need to click the button to display the remainder. Because this feature can annoy readers, many content providers also display a “turn off reveal full article” button as an alternative.

Jon Kleinberg’s HITS (hyperlink-induced topic search) algorithm has become a popular and more effective way to rate web pages.^[6] HITS ranks documents by the links within the document, presuming that a good document incorporates and references (provides links to) other web documents while also being frequently cited by other documents. Hubs—documents with many links—are related to authority pages, or frequently cited documents. This relationship of hubs and authority is mutually reinforcing; if you imagine a Web universe of one hundred pages, the one with the most links and most frequent references wins.

As a business writer, you’ll naturally want to incorporate authoritative sources and relevant content. But you’ll also want to attract and engage your audience, positioning your document as a hub and authority within that universe. Feedback in the form of links and references may be one way to assess your online document.

User-Generated Feedback

Moving beyond web tracking aspects of feedback measurement, let’s examine user-generated responses to your document. Say you’ve reviewed the posts left by unique users in the comments section of your article. This serves a similar purpose to letters to the editor in traditional media. In newspapers, magazines, and other offline print media, an edition is produced with a collection of content and then delivered to an audience. The audience includes subscriber-based members with common interests, as well as those who read a magazine casually while waiting in the doctor’s office. If an article generated interest, enjoyment, or outrage—or demanded correction—people would write letters in response. Select responses would be published in the next edition. There’s a time delay associated with this system that reflects the preparation, production, and distribution cycle of the medium. If the magazine is published monthly, it takes a full month for user feedback to appear in print—letters commenting on an article in the March issue would appear in the “Letters” section of the May issue.

With online media, the speed of this feedback loop has greatly increased. Public relations announcements, product reviews, and performance data are often made available internally or externally via electronic communication. If you spot a factual error in an article released internally, you may be able to respond within minutes with an email and attachment correcting the data. Similarly, if the document is released externally, expect quick feedback from outside your organization. Audience members may debate your description of the water pump, openly question its effectiveness in relation to its specifications, or post positive comments. Customer comments, like letters to the editor, can be a valuable source of feedback.

Customer reviews and similar forms of user-generated content are increasingly common across the Internet. Written communication is often the preferred format; from tweets to blogs and commentary pages, to threaded theme-based forums, person-to-person exchange is increasingly common. Still, as a business writer, you’ll note that even with the explosion of opinion content, the tendency for online writers to cite a web page with a link can and does promote interaction.

It may sound strange to ask this, but is all communication interaction good? Let’s examine examples and see if we can arrive at an answer.

You may have heard that one angry customer can influence several future customers, but negative customer reviews in the online information age can make a disproportionate impact in a relatively short time. While the online environment can be fast and effective in terms of distribution and immediate feedback, it can also be quite ineffective, depending on the context. “Putting ads in front of Facebook users is like hanging out at a party and interrupting conversations to hawk merchandise,” according to Newsweek journalist Daniel Lyons.^[7] Relationships between users, sometimes called social graphs, reflect the dynamic process of communication and hold value, but translating that value into sales can be a significant challenge.

Overall, your goal as a business writer is to meet audience and employer expectations in a clear and concise way. Getting your content to a hub position and including authoritative references is a great way to make your content more relevant. Trying to facilitate endless discussions may be engaging and generate feedback, but it may not translate into success. Facebook serves as a reminder that you want to provide solid content and attend to feedback. People who use Google already have something in mind when they search; if your content provides what readers are looking for, you may see your page views and effectiveness increase.

Obtaining Feedback with Qualitative Research

Qualitative research involves investigative methods that cross subjects and academic disciplines to gain in-depth information. If quantitative research explores “what,” qualitative research explores “how” and “why.” From interviews to focus groups, many face-to-face strategies for gathering information are qualitative in nature.

You have five senses, and you may be able to distinguish between sweet and salty foods, but can you describe what you taste and smell? Let’s say you work for a vineyard and have been tasked to write a paragraph describing a new wine. Could you? Capturing fine data points and representing them in words and symbols can be a significant challenge for researchers. When testing the wine with a focus group, you might want information on how it’s perceived, and responses may be varied and unusual. What do you do with the information you gather? You may identify trends among varied responses and create groups indicating a woody or earthy flavor. Still, numbers fail to capture the nuances of flavor and body.

Some information—like the way consumers characterize a wine’s taste—is a challenge to obtain, and qualitative research often serves well here. Where quantitative research is effective with large audiences, qualitative research allows for in-depth interpersonal interviews that produce rich and meaningful results. The information may not be as reliable, and your ability to produce the same results over time may be limited, but humans are emotional, irrational, and unpredictable. They’re also, each in their own way, unique. As you increase the level of abstraction, all humans may eventually come to look similar, even the same. We all share some characteristics, such as the use of language or the composition of our bodies. But look more closely and you see the diverse range of languages, and learn that not everyone has 206 bones in an adult body. Between these two views, we find the range of information that quantitative and qualitative research attempt to address.

Suppose we want to determine who has a greater lifetime risk of developing heart disease—a man or a woman? If we’re talking about an individual man and an individual woman, our answer might differ from what it would be if we were talking about men and women in general. A survey may work well to capture data about men versus women broadly, but a face-to-face interview with a man and a woman allows for interaction, follow-up questions, and a much better picture of the question: between this individual woman and this individual man, who is more likely to be at risk? The risk and protective factors we learn from broad research projects involving thousands of subjects have value, but sometimes a broad brushstroke fails to capture the fine data needed or desired.

Imagine you’re involved in direct observation of buying behavior by reviewing security camera videos that clearly show your company’s product in relation to other products on the shelf. You may find—particularly after a review of the literature—that product placement makes a significant impact on purchase decisions. You may also be involved to some extent in the setting. Serving as a participant observer means you’re part of the process, involved in action, and not separated from the interaction. You look at the sales experience through the eyes of a participant and view others through the eyes of an observer. You may find that interviews and focus groups teach you more about your audience, but you may also learn from others who have conducted similar interviews.

As a business writer, you should be familiar with qualitative research and its relative strengths and weaknesses. You may use some of its techniques to gather information about your audience, cite research that involves qualitative methods, and use its strategies with an audience after document, product, or service delivery.

Obtaining Feedback with Quantitative Research

Quantitative research involves investigation and analysis of data and relationships between data that numbers can represent. It’s often used to test a hypothesis and normally involves large volumes of data. Where a qualitative research project might involve a dozen interviews, a quantitative one would involve hundreds or thousands. Since each interview carries a cost—and a thousand or ten thousand interviews may exceed your organization’s research budget—a more cost-effective alternative must be found. By limiting the number of questions and the ways in which participants can respond, data can be gathered at lower cost, often with a higher level of statistical validity.

In qualitative research, you may ask an open-ended question like “What does the wine taste like?” In quantitative research, you may limit the response options: “Does the wine taste (a) woody, (b) fruity, or (c) both?” You may find that 90 percent of respondents indicate answer (c); you can represent it with numbers and a graph, but it may not serve your investigation the way you planned.

Research methodologies examine and evaluate the methods used in investigation or soliciting feedback. They address and improve poorly worded questions and help the investigator match the research goal to the method. Quantitative research serves us well when we ask, “Does vitamin C, taken at a dosage of 500 mg daily for five years, lower the incidence of the common cold?” We could track a thousand participants who provide intake prescreening information, confirm daily compliance, and participate in periodic interviews. We also know that part of our group is taking a placebo (sugar pill) as part of the requirements of a double-blind study. At the end of the term, we have numbers that may indicate the degree to which vitamin C affects illness incidence.

Advertisers often research to learn more about preferences and attitudes—two areas not easily captured. Sometimes preference studies use Likert scales, which give respondents a preset scale to rate their answers. An example of a Likert item might be: “Please indicate to what degree you agree or disagree with this statement: I enjoy drinking brand X wine. Do you (1) strongly agree, (2) agree, (3) neither agree nor disagree, (4) disagree, or (5) strongly disagree?”

There’s a tendency for some attitudinal and preferential research that may be more accurately described as qualitative to be presented in numerical terms. For example, you’ve probably heard the claim that “four out of five dentists prefer brand X,” when in itself, the number or representation of preference is meaningless. As an astute business writer, you’ll understand pre (before) and post (after) document, product, or service research investigations and distinguish between the two main approaches.

What Is Validity?

How do you know the results presented in a study or article have value? How do you know they’re valid? Validity involves the strength of conclusions, inferences, or assertions. Thomas Cook and Dan Campbell indicate that validity is often the best available approximation of the truth or falseness of an inference, proposition, or conclusion.^[9] Readers want to know your information has value and that there’s confidence in its points, supporting information, and conclusions. They want to know you’re right and not making false statements.

One way to address validity is to cite all your sources clearly. As a writer, you can certainly include information from authorities in the field when the attribution is relevant and the citation is clear. Giving credit where credit is due makes your information more valuable, and by referencing sources clearly, you enable readers to assess the validity of your information.

Does all feedback have validity? Just as there are many threats to validity in research applications, you can’t always be sure that feedback is accurate or truthful. Have students ever evaluated professors negatively because of the required work in the course? Of course. Similarly, some readers may have issues with the topic or your organization. Their feedback post may be less than supportive, even openly hostile. Assess the validity of the feedback, respond with professionalism at all times, and learn to let go of negative messages that offer little opportunity to improve understanding.

What Is Reliability?

Reliability is the consistency of your measurements. The degree to which an instrument gives the same measurement each time with the same subjects, in the same context, is a measure of its reliability. For example, if you took your temperature three times within fifteen minutes and your thermometer gave a different reading each time—say, 98.6, 96.6, and 100.2—you’d conclude that your thermometer was unreliable.

How does this apply to feedback in business writing? Let’s say you have three sales agents who will complete follow-up interactions with three customers after you’ve sent a report to each customer on their purchases to date with suggestions for additional products and services. All three sales agents have the same information about products and services, but will they perform the same? Of course not. Each one, even if trained to stay on script and follow specific protocols, won’t be identical in approach and delivery. Each customer is also different, so the context is different in each case. As business professionals, we need to learn about our environment and adapt to it. This requires feedback and attention to information in many forms. We need to assess the strength or weakness of information, its reliability or validity, and be prepared to act on it. Successful businesses—and by extension successful business communicators—recognize that communication is a two-way process in which we need to listen, learn, and respond to feedback. We need to meet and exceed customer expectations.

Inter-rater reliability involves the degree to which each evaluator evaluates the same in similar contexts. Consider a college essay to better understand this concept. Let’s say you write an essay on customer relationship management and submit it to your business communication instructor. At the same time, you submit the same essay to your English professor and your marketing professor. Will all three professors evaluate your essay the same? Of course not. They’ll each have their own expectations and respective disciplines that influence what they value and how they evaluate. Still, if your essay is thoroughly researched, logically organized, and carefully written, each professor may give it a better-than-average grade. If this is the case, inter-rater reliability would indicate that you did a good job on the essay.

What Is Statistically Significant?

This research term is often used and commonly misunderstood. Not every research finding is statistically significant, and many that are considered significant are only slightly more likely than pure chance. Statistically significant findings are those with a high level of reliability—if the same test is applied in the same context to the same subjects, the results come out the same time and time again.^[10] You may see a confidence level of +/– (plus or minus) three percentage points as a common statement of reliability and confidence in a poll. It means that if the poll were repeated, there’s confidence that results would be within three points above or below the original percentages. When statements of statistical significance are made, you’ll know that a difference or relationship was established with confidence by the study. That confidence gives the results credibility.

All of this—qualitative versus quantitative, validity, reliability, statistical significance—is really one skill wearing four different hats. The skill is reading feedback carefully enough to know what it’s actually telling you. A qualitative focus group tells you the texture of how a specific handful of people responded. A quantitative survey tells you the rough shape of how a larger group responded. Validity tells you whether the thing being measured is the thing you think is being measured. Reliability tells you whether you’d get the same measurement if you looked again. Statistical significance tells you whether a difference is big enough to be worth caring about. None of the four hats, by itself, is enough to make a decision. All four together start to give you something you can act on.