AI-powered workflow · MHCI capstone

Goodwill Agent

Describe a garment while holding it, and its tag prints right at your station.

  1. Describe
  2. Check & confirm
  3. Print the tag

Goodwill’s existing POS system

The voice strip we designedRecognized · Ready to confirm

The voice strip before confirmation. The transcript ‘Womens long sleeve blouse, medium, better’ has filled Division, Category, Size and Price. The green Confirm button is ready to submit the item.

Static preview of the simulation; item details and prices are examples.

Try the interactive prototype

The voice strip we designed

Swipe or scroll sideways to see the full strip.

The horizontal voice strip, with Listening on the left, recognized item fields and their transcript in the middle, and Clear and Confirm on the right. All controls keep their original positions.
My role
Product Design + Tech LeadI led the user research, product design, and engineering.
Client
Goodwill
Team
5 CMU MHCI graduate students
Responsibilities
Contextual inquiry · Wizard-of-Oz study · Data analysis · Voice workflow design · Windows automation engineering

Project outcomes

+112%soft-line items a processor handled per hour (about 39 → 83), in a one-week in-store pilot
3 → 1times each item is handled, with tag-as-you-go
1stinternal AI workflow tool at Goodwill
0 → 1field research → design → shipped code

The challenge

01 / 06

Slow processing leaves clothes stuck in the back room

Slow back-room processing keeps donated clothing off the sales floor. We set out to find the bottlenecks in softlines (clothing and household textiles) and make tagging faster and less demanding.

$23.66Mannual softlines sales across Goodwill SWPA’s 32 stores
~50%annual processor turnover, so a new workflow has to be quick to learn
A Goodwill back room: pallets stacked with sealed boxes, open cardboard bins and blue bins overflowing with donated clothing and household items, and a loading door, empty hanger racks and flattened cardboard behind them.
Donated clothing waits in the back room before it can reach the sales floor.

Spring research

02 / 06

We did the job ourselves before proposing anything

We watched processors work and explain their decisions, then processed donated garments ourselves. Doing the job made the repeated walking, remembered prices and time pressure tangible.

A person in a red Carnegie Mellon sweatshirt and blue gloves operates the Solutions POS touchscreen at a store workstation, with a label printer beside it.
Using the store’s POS system ourselves to experience the logging and tagging workflow.
4store visits
8processor interviews
8think-aloud sessions
36+donated garments we processed ourselves

One rack, five places the current workflow loses time

Once a rack was full, processors pushed it to a shared POS, waited for a computer and entered every garment before printing tags.

The current workflow, as a floor map Schematic of the back room: three processing stations along the bottom, the shared POS at top left, hardlines at top right, and the exit to the store floor in the top left corner. One rack goes from Station 3 to the POS to be logged and tagged, then to the store floor. Numbers 1 to 5 mark the five places the workflow loses time. Store floor Shared POS Hardlines Station 1 Station 2 Station 3 1 2 3 4 5
A schematic of an observed back room: racks pass through the shared POS before reaching the sales floor.
  1. Cognitive load

    Prices are decided at the rack but must be remembered until logging.

  2. Waiting for a shared desktop

    Up to four processors share two desktops, creating a queue.

  3. Travel across the floor

    Every full rack travels to the POS, taking time and aisle space.

  4. A hard-to-use POS

    Logging each garment takes several levels of menu selections.

  5. Handling each item three times

    Sorting, logging and tagging each require picking up the same garment.

Keep processors at the rack

  1. Problem statementReduce cognitive load and physical effort while helping processors tag clothing faster and more accurately.
  2. DirectionDesign a hands-free workflow that lets processors enter item information while staying at the rack.
Later in-store pilot footage (46 seconds, with sound): logging and tagging without leaving the rack.

Summer research

03 / 06

Three choices shaped the final design

Summer testing centered on three choices: how processors would enter information, where the interface would sit, and whether to work one item at a time or in batches.

9store visits
15research experiments
18prototype iterations
23 hoursof in-store pilot testing

Why voice? Keep hands on the garment

In storyboard comparisons, processors rated voice lowest in mental effort and less physically demanding than tapping; smart glasses ranked last. In timed voice-vs-touch tasks, most were faster with voice and preferred it.

We tested mic and transcription-tool combinations in an echoing basketball court to check performance in noise.

97–98%voice transcription accuracy with a clip-on mic, in a noisy test environment
12mic and transcription-tool pairings tested
4processors compared three hands-free concepts in storyboards
5processors ran timed voice-vs-touch tagging tasks
More detail: speech model comparison

Team members read Goodwill’s pricing-category vocabulary to compare cloud and local speech models. We chose Deepgram for accuracy, paired with Vosk to keep working offline.

97.6%Deepgram (cloud, chosen): 1.5 errors on average across 63 terms
95.2%Azure (cloud): 3.0 errors on average across 63 terms
90.8%Vosk (local, chosen): 6.75 errors on average across 73 terms
53.4%Sherpa (local): 34.0 errors on average across 73 terms

Why a voice strip?

We explored full-screen layouts, a separate app and an overlay. Solutions also handled inventory, checkout and production, so it could not be replaced and had no public API. A browser proof of concept established automated clicking; the interfaces then had to share one screen.

Early full-screen wireframe: logged items on the left, an expanded category grid on the right, division and size above, and voice and print buttons below.
Full-screen wireframeReorganized categories, sizes and logged items, but data still had to pass through Solutions.
Separate tablet app: logged items on the left with prices blurred, item fields and a transcript in the middle, and action buttons on the right.
Separate tablet appBrought voice and item details together, but switching apps interrupted Solutions automation.
Final voice strip floating over the bottom of Solutions, with the existing product buttons still visible.
Floating voice stripKept Solutions full-screen, with input, review and confirmation in one overlay.

The wireframe is from the team’s Figma file; the other images show prototypes. Open any image to view it at full size.

More designs: layout, expanded and collapsed strips

The full-screen wireframes explored an open category grid and a layout focused on the current selection. Widget designs also explored expanded and collapsed forms; design notes discussed button placement, listening status and transcript size.

Alternative full-screen wireframe: the item list stays on the left, while the right side focuses on Jacket and expands Good, Better, Best and Open options.
Another full-screen layout: collapse categories and enlarge the current selection.
Expanded voice-strip draft with paused status, item fields, quantity, count and confirmation arranged horizontally.
Expanded option: fields and actions stay visible together.
Collapsed voice-strip draft with a resume button and listening status in a small floating window.
Collapsed option: a small window keeps the microphone controls.
Try the earlier standalone tablet prototype

Why tag as you go? Finish each item once

Logging and tagging each garment on the spot reduced repeated handling.

25% less time20-item comparison: about 10 → 7.5 minutes
3 → 1times each item is handled

Solution

04 / 06

Goodwill Agent turns a spoken description into a printed tag

The voice strip floats over Solutions, turning a description into item details the processor can check before the agent logs them.

The new workflow, as a floor map The same back room: every station has a tablet and a printer, processors wear headsets, and the shared POS is no longer needed. The rack goes straight from Station 3 to the store floor. Numbers 1 to 4 mark the four parts of the new workflow. Store floor Shared POS No trips needed Hardlines Station 1 Station 2 Station 3 1 2 3 4
The same back room, redesigned: racks go directly from the station to the sales floor.
  1. A tablet at every station

    A tablet and printer beside the rack remove trips and queues.

  2. Hands-free voice input

    Processors describe garments while keeping their hands on them.

  3. The agent operates Solutions

    The agent clicks through the existing software for the processor.

  4. Tag as you go

    Log, print and attach the tag before starting the next garment.

Say, see, tap: three channels, one job each

Voice handles input. The screen shows the transcript and interpreted fields. Touch handles confirmation and corrections, keeping the final decision with the processor.

Goodwill Agent voice strip docked over the Solutions DGR POS screen: a pause button and “Listening” status on the left; department Womens, category Long Sleeve Blouses, size 2XL+ and price tier Better $6.40 across the middle; quantity, a daily count of 256, and the Clear and Confirm buttons on the right; the live transcript along the bottom.
The final voice strip, numbered to match the notes below.
  1. Listening · Pause

    The waveform shows mic activity; a button or spoken command pauses and resumes it.

  2. What it heard

    The separate transcript helps distinguish mishearing from misinterpretation.

  3. What it understood

    Department, category, size and price tier stay visible and editable.

  4. Confirm · Clear

    All four fields are required. Logged confirms success; Clear starts over.

  5. Quantity · Daily count

    Adjust quantity as needed; successful entries add to the daily count.

  6. Solutions behind it

    The agent’s clicks stay visible so processors can follow its progress.

Interactive prototype

Tap the example below or type an English description, check the item details, then tap Confirm to watch the agent log it.

This interactive prototype needs JavaScript. Watch the in-store pilot footage in “Keep processors at the rack” to see the same flow in real use.

Keep the processor in control

I built the agent as a Windows app running on the station’s PC. It checks each step before proceeding and stops for a person whenever the screen differs from what it expects.

More detail: engineering and deployment validation

The agent uses the processor’s existing session without accessing their Solutions credentials. Commands are idempotent and run one at a time. Every step goes through a verify-act-verify state machine, which stops and hands over to a person whenever the screen doesn’t look as expected.

In our survey of 131 Goodwill employees, only 8% trusted AI recommendations. In an organization like that, these invisible constraints mattered as much as the confirmation on screen. Whenever the system isn’t sure, control goes back to a person.

After two on-site visits with Goodwill IT, the agent ran the full tagging workflow on the live Solutions system (DGR.exe) in Goodwill’s controlled test environment. It was delivered as a self-updating installer.

Goodwill Agent: one Windows appalways-on-top voice strip · runs locally on the POS computer · no server, no pairing
Voice & command pipelinevoice-activity-gated mic · switchable engines (offline or cloud) · apparel vocabulary with fuzzy + phonetic matching · idempotent sequential queue · verify-act-verify UI Automation state machine
SolutionsGoodwill’s existing POS system, which logs the item and prints the tag

Pilot and impact

05 / 06

In the pilot, items per hour more than doubled

One processor used the tool for one work week, so this is directional evidence; a larger sample is still needed.

“I love it. This is great, this is going to make us faster.”

Pilot processor
+112%soft-line items a processor handled per hour, in a one-week in-store pilot
39 → 83items per hour, against that processor’s six-month average
More detail: rollout costs and revenue projections

These are projections, not realized results. Four scenarios combine production gains of 50%–112% with different sell-through rates, projecting $4.7M–$26.5M in added annual revenue. The conservative case assumes a 50% gain and 45% sell-through.

Annual running costs include tablets, headsets and printers spread over three years, speech-model usage, IT maintenance and a 10% contingency.

+$4.7Mprojected added revenue a year, in the most conservative scenario
$89.8Kestimated annual cost to run it across the region
0.38%of current annual softlines sales
7 daysto recover a year’s running cost, in the most conservative scenario

What’s next

06 / 06

Four things to test next

  1. 01A larger pilot

    Include more processors across different stores in daily use.

  2. 02More accents and languages

    Test more accents and voice input and translation for Spanish and other languages.

  3. 03More categories

    Make product categories and prices configurable.

  4. 04Feedback that keeps up with trust

    Pilot processors quickly trusted the system and stopped checking the screen. Feedback must still help them catch mistakes.

Next case study

Social networking app · CMU course project Hive Five Group meetups for Bumble BFF