Technology products are embedded in every aspect of daily life from homes, cars, phones, schools, workplaces. They’re in entertainment, healthcare, safety, and beyond. While technology is often billed as making things easier, faster, cheaper, and fairer, it can cause harm at scale. People face frustration, harassment, financial loss, physical harm, and more.
What are “Flawed products?” Flawed products are products, services, and technologies developed without considering, including, and understanding the needs of underserved consumers expected to buy and use them.
I use the term “flawed” because if we are practicing user centered design, human centered design, customer centricity, design thinking, or similar methodologies we need to root our research and design in understanding and meeting the needs of users. If users are not being considered, included, and understood, there is a flaw.
With the historical and present-day pattern of creating products with little or no consideration of underserved groups, deliberate and sustained focus on designing products that meet the needs of underserved groups is an imperative. Flawed products particularly result in harm to racial minorities, the poor, the disabled, and in some cases women.
Negative Impacts of Flawed Products
- Consider one follow up care recommendation tool used by hospitals that research showed inaccurately classified 82% of white patients as needing follow up care vs 18% of black patients that were just as sick or sicker.1 This flawed tool used insurance claim data instead of the patient’s actual health data, ignoring well-known health disparities in the US healthcare system.
- Slack Connect was designed initially to allow Slack users to send messages to users in other Slack groups as long as they had their email address. Users did not have an opt out option and the ability to customize messages coming from a generic email address left them vulnerable to harassment, spamming, or phishing. In online spaces where women and minorities are often the targets of online harassment, this was a problematic design.
- Research on facial recognition software has shown it misidentifies people of color at much higher rates than whites. Yet, it continues to be used in law enforcement, recruitment, education, and unemployment benefits decisions. Despite facial recognition software leading to the false arrest of several black males, police departments are still using them.2
The Increasing Backlash
With social media and rapid news cycles amplifying the stories of people who have been negatively impacted by flawed products, consumers are demanding stronger protections against flawed technology products. To date a small number of state and local governments have banned harmful products like facial recognition software.4
Companies are also being pressured internally by employees and externally by advocacy groups and investors to reign in technology products. This push has strengthened and become intertwined in the broader racial justice and economic justice movements over the last few years, requiring individuals, teams, and companies to design products better.
How to ensure that you aren’t making flawed products?
We need a framework to proactively address common issues with product design and flawed products:
- Lack of diverse input in the product design and/or testing
- Ignoring how historical and present-day racism, sexism, classism, ableism, etc. relates to their product
- No examination of what harm the product could create or how it could be misused
- Lack of consideration of what could go wrong within the context of customers’ lives
- Lack of accountability for the problems the product created
- Poor implementation and poor problem resolution
In 2019, I created the “Three Questions Do No Harm Framework” or “3Q-DO NO HARM Framework” to help individuals, teams, and organizations identify and mitigate potential harms before a product was released.
While the framework is deceptively simple with just three questions, the hard work is to not just ask the questions but answer them and act on the issues identified to avoid or mitigate harm. The 3Q-DO NO HARM Framework can be applied on the individual, team, or organizational level.
Applying the 3Q-DO NO HARM Framework
Question 1: Who’s not here?
To actively identify and resolve your knowledge gaps, ask research, design, and development teams as well as research participants, “Who’s not here?” A lack of awareness, understanding, and representation of different groups and their needs contributes to flawed products that create problems.
Actions to Take:
- Expand team diversity by eliminating hiring barriers such as the over-reliance of employee referrals or cultural fit that perpetrate hiring people similar to the current team.
- Conduct research with underrepresented groups to develop a nuanced understanding and avoid stereotypes.
- Expand your research pool by ensuring research opportunities are accessible. Offer research times, locations, and methods convenient for a wide range of users.
It’s the responsibility of all team members to be aware, knowledgeable, and focused on inclusive research.
Question 2: How will vulnerable groups be negatively impacted?
Identify unintended consequences and mitigate beforehand, designers must actively consider what harm their product might do particularly for vulnerable groups.
Actions to Take:
- Use a systems approach to map the impact of your product. Take a broad look at the context as well as core elements, interconnections, and structures that your product will operate in. Brainstorm what would go wrong and how your product could be misused. For example, an online communications platform for a neighborhood may make it easier to share relevant news, it can also exerbate tensions between neighbors when larger systemic issues like gentrification, racial profiling, and redlining aren’t considered.
- Discover and consider the constraints vulnerable groups live under. For example, consider whether a product change requires time, money, technology, transportation that some users may have little or no access to.
Take corrective action to eliminate or mitigate any harms before the product is launched.
Question 3: When things don’t work, how will it be quickly resolved?
Planning for the unhappy path by ensuring the path to resolving problems is clear and fast. While the intent is to design so everything works perfectly, technology often doesn’t work as designed for all users.
Actions to Take:
- Incorporate into your research questions on how users have resolved issues with the technology in the past. Find out what did and didn’t work. Integrate solutions into your design.
- Advocate for a role in implementation and continue to monitor and iterate to take the onus of customers to resolve problems.
The “3Q-DO NO HARM Framework offers a simple structure to help individuals, teams, and organizations to take action to avoid the harmful effects of poor product decisions. There’s an urgency brought on by consumers, advocacy groups, and governments demanding companies provide better products and eliminate harmful patterns.
1 https://www.statnews.com/2019/10/24/widely-used-algorithm-hospitals-racial-bias/
2 https://www.buzzfeednews.com/article/ryanmac/facial-recognition-local-police-clearview-ai-table
3 https://www.nature.com/articles/d41586-021-00868-5
4 https://edition.cnn.com/2021/05/18/tech/amazon-police-facial-recognition-ban/index.html
Featured photo by Headway on Unsplash.