How Coffees Get Scored

Do you know how coffees get scored?

Here’s a sample espresso cupping form:

Let’s talk survey methodology.

This form is part of a concerted effort to make expert coffee tasters’ assessments more scientific and objective, but it’s ultimately based on a rather cartoonish idea of how science actually works.

Here’s the heart of the “scientific” methodology at play:

step 1: break the whole up into pieces

step 2: rate each piece individually

step 3: add the individual ratings back together for a total rating

Let’s start by saying something nice about each step.

step 1: break the whole up into pieces

• This is a solid strategy to try to identify different aspects of coffee.

• (Even if it doesn’t seem to recognize how interrelated they are.)

step 2: rate each piece individually

• This forces tasters to pay attention to subtle differences along each dimension of value.

• (By circling different numbers.)

step 3: add the individual ratings back together for a total rating

• This provides a consistent procedure for assessing every coffee.

• (By making +1 Aroma just as good as +1 Body, whatever that means.)

Before we start reading individual scales, look how they’re framed visually.

6,0 or below is gray—insufficient.

6,5 or above is white—sufficient or better.

Under the gray, we ask for Unpleasant perceptions.

Under the white, we ask for Pleasant perceptions.

Oh yeah.

This is an incredibly binary, good/bad notion of tasting.

One limitation of exclusively quantitative tasting is that it tends to reduce one’s tasting to liking and not liking. Kenneth Liberman (p. 259)

Seriously, let’s take a look at this form. I’m just reading straight through.

Cream has two parts of equal importance:

1. Colour (from no to striped)

2. Aspect/Persistency (from very poor to very compact/longlasting)

3. Next is Aroma (from very defective to outstanding) which also gets to include a blurb for any Pleasant/Unpleasant perceptions.

4. We rate Bitterness (quality) from very bad to outstanding.

5. And Acidity (quality) from very aggressive to outstanding.

6. Then Flavour from very bad to outstanding along with another Pleasant/Unpleasant blurb. (I guess Flavour’s just obviously a single quality?)

7. Sweetness from absent to outstanding.

8. Body (quality) from very dry to outstanding.

9. Aftertaste from very unpleasant to outstanding, with an additional Pleasant/Unpleasant blurb.

And then in what is definitely just a coincidence and not what Liberman roasts as a “digital fetish,” there’s a tenth one:

10. Overall balance from absent to outstanding.

Alright now sum it up, deduct any penalties, and you’re done!

Each aspect is rated 1 to 10 so yes, a perfect score would add up to 100. =]

Thanks to Science, we’ve discovered whether your coffee is barely adequate, good, remarkable, excellent, or outstanding!

Obviously I have tons of nitpicky questions here:

• I’m very intrigued by which aspects bottom out at totally absent as opposed to very X.

  • (What separates those that admit of total absence and those that don’t?)

• Also I take it very poor and very bad are different enough to use both? What’s up with that?

  • (In the Italian: molto carente vs. molto sgradev.)

• How many different opposites does outstanding have anyway?!

But here’s what I find really interesting...

It turns out that after you bring expert tasters together and let them norm and calibrate together for a short period of shared tasting, they do a remarkably good job of rating coffees consistently, even though there are several chemical reasons that apparently make tasting coffee trickier and subtler than tasting wine.

If you give these expert tasters the “same” cup twice, it’s never really the same—the process of preparing coffee is incredibly sensitive down to the level of individual bad beans spoiling the lot, tiny roasting inconsistencies, and countless other variables.

Even so, expert tasters re-rate that “same” cup remarkably close to the original. Remember, they just have to be consistent enough for the economics to work, for everyone to be confident enough in the quality of coffee they’re dealing with, with an emphasis on avoiding catastrophe. No one wants a bad batch disappointing their customers!

And yet, while much day-to-day tasting work is tasting for defects, these experts are also refined enough to taste for the very highest quality.

It’s largely because of the form!

But not how you think.

Here’s their basic process:

1. Tasters scribble copious notes trying to articulate what they’re discovering in the coffee as it changes and cools over time.
   

2. Then they translate their expert assessments into data—one of fourteen numbers from 1 to 10 for each of the ten aspects listed, and a handful of short blurbs.
   

3. And then a powerful algorithm (called Addition) scores each coffee.

During steps 1 and 2, the form is very helpful for providing structure and routine to tasters’ inquiry.

But when tasters find that the mechanical sum over- or underrates a coffee, step 4 is to bust out the eraser.

This 84 should really be closer to 87, so let me add a point to Aroma but hmmm, can I really upgrade it from quite compact to compact/longlasting?

You know, I thought the last coffee was a 10 on Body but really this one’s a 10, the last one’s 9.5...

It’s in these moments, when these experts recognize the limitations of their tools and “game the chart,” that Kenneth Liberman thinks they are being the most objective—in a certain sense of objective!

It’s a very interesting claim.

He goes, this is when they are most attuned to the object at the expense of their methodology—you know, the most object-ive, as opposed to robotically methodical—which is just as it should be in proper scientific work that’s alive to the limitations and contingencies of its own current methodologies!

By letting the coffee speak over the statistical artifacts of their survey tools, tasters do a better job of assessing the coffee’s overall quality in a consistent way, which was the whole point of this expensive tasting practice anyway!

Wanna read more?

Check out Tasting Coffee: An Inquiry into Objectivity, by Kenneth Liberman.

This is the first work of ethnomethodology I’ve read and I wanna check out more to figure out what’s going on.

In the past, I’ve really enjoyed diving into anthropology and sociology, but this takes such a close look at the microsocial, the meaningful looks and tiny hedges (“For me, …”) that tasters use in the process of arriving at consistent, objective ratings because it turns out everyone’s a little shy about sticking their neck out.

Is everyone in ethnomethodology doing this much phenomenology?

Stay tuned as my kaleidoscopic tumble through philosophy of science continues!