Tag: Conversion

22 Jun

Image To Text Conversion With React And Tesseract.js (OCR)

by TBSCategories: News

Data is the backbone of every software application because the main purpose of an application is to solve human problems. To solve human problems, it is necessary to have some information about them.

Such information is represented as data, especially through computation. On the web, data is mostly collected in the form of texts, images, videos, and many more. Sometimes, images contain essential texts that are meant to be processed to achieve a certain purpose. These images were mostly processed manually because there was no way to process them programmatically.

The inability to extract text from images was a data processing limitation I experienced first-hand at my last company. We needed to process scanned gift cards and we had to do it manually since we couldn’t extract text from images.

There was a department called “Operations” within the company that was responsible for manual confirming gift cards and crediting users’ accounts. Although we had a website through which users connected with us, the processing of gift cards was carried out manually behind the scenes.

At the time, our website was built mainly with PHP (Laravel) for the backend and JavaScript (jQuery and Vue) for the frontend. Our technical stack was good enough to work with Tesseract.js provided the issue was considered important by the management.

I was willing to solve the problem but it was not necessary to solve the problem judging from the business’ or the management’s point of view. After leaving the company, I decided to do some research and try to find possible solutions. Eventually, I discovered OCR.

What Is OCR?

OCR stands for “Optical Character Recognition” or “Optical Character Reader”. It is used to extract texts from images.

The Evolution Of OCR can be traced to several inventions but Optophone, “Gismo” , CCD flatbed scanner, Newton MesssagePad and Tesseract are the major inventions that take character recognition to another level of usefulness.

So, why use OCR? Well, Optical Character Recognition solves a lot of problems, one of which triggered me to write this article. I realized the ability to extract texts from an image ensures a lot of possibilities such as:

  • Regulation
    Every organization needs to regulate users’ activities for some reasons. The regulation might be used to protect users’ rights and secure them from threats or scams.
    Extracting texts from an image enables an organization to process textual information on an image for regulation, especially when the images are supplied by some of the users.
    For example, Facebook-like regulation of the number of texts on images used for ads can be achieved with OCR. Also, hiding sensitive content on Twitter is also made possible by OCR.
  • Searchability
    Searching is one of the most common activities, especially on the internet. Searching algorithms are mostly based on manipulating texts. With Optical Character Recognition, it is possible to recognize characters on images and use them to provide relevant image results to users. In short, images and videos are now searchable with the aid of OCR.
  • Accessibility
    Having texts on images has always been a challenge for accessibility and it is the rule of thumb to have few texts on an image. With OCR, screen readers can have access to texts on images to provide some necessary experience to its users.
  • Data Processing Automation The processing of data is mostly automated for scale. Having texts on images is a limitation to data processing because the texts cannot be processed except manually. Optical Character Recognition (OCR) makes it possible to extract texts on images programmatically thereby, ensuring data processing automation especially when it has to do with the processing of texts on images.
  • Digitization Of Printed Materials
    Everything is going digital and there are still a lot of documents to be digitized. Cheques, certificates, and other physical documents can now be digitized with the use of Optical Character Recognition.

Finding out all the uses above deepened my interests, so I decided to go further by asking a question:

“How can I use OCR on the web, especially in a React application?”

That question led me to Tesseract.js.

What Is Tesseract.js?

[Tesseract.js is a JavaScript library that compiles the original Tesseract from C to JavaScript WebAssembly thereby making OCR accessible in the browser. Tesseract.js engine was originally written in ASM.js and it was later ported to WebAssembly but ASM.js still serves as a backup in some cases when WebAssembly is not supported.

As stated on the website of Tesseract.js, it supports more than 100 languages, automatic text orientation and script detection, a simple interface for reading paragraphs, words and character bounding boxes.

Tesseract is an optical character recognition engine for various operating systems. It is free software, released under the Apache Licence. Hewlett-Packard developed Tesseract as proprietary software in the 1980s. It was released as open source in 2005 and its development has been sponsored by Google since 2006.

The latest version, version 4, of Tesseract was released in October 2018 and it contains a new OCR engine that uses a neural network system based on Long Short-Term Memory (LSTM) and it is meant to produce more accurate results.

Understanding Tesseract APIs

To really understand how Tesseract works, we need to break down some of its APIs and their components. According to the Tesseract.js documentation, there are two ways to approach using it. Below is the first approach an its break down:

Tesseract.recognize(   image,language,   {      logger: m => console.log(m)    } ) .catch (err => {   console.error(err); }) .then(result => {  console.log(result); }) }

The recognize method takes image as its first argument, language (which can be multiple) as its second argument and { logger: m => console.log(me) } as its last argument. The image format supported by Tesseract are jpg, png, bmp and pbm which can only be supplied as elements (img, video or canvas), file object (<input>), blob object, path or URL to an image and base64 encoded image. (Read here for more information about all of the image formats Tesseract can handle.)

Language is supplied as a string such as eng. The + sign could be used to concatenate several languages as in eng+chi_tra. The language argument is used to determine the trained language data to be used in processing of images.

Note: You’ll find all of the available languages and their codes over here.

{ logger: m => console.log(m) } is very useful to get information about the progress of an image being processed. The logger property takes a function that will be called multiple times as Tesseract processes an image. The parameter to the logger function should be an object with workerId, jobId, status and progress as its properties: 

{ workerId: ‘worker-200030’, jobId: ‘job-734747’, status: ‘recognizing text’, progress: ‘0.9’ }

progress is a number between 0 and 1, and it is in percentage to show the progress of an image recognition process.

Tesseract automatically generates the object as a parameter to the logger function but it can also be supplied manually. As a recognition process is taking place, the logger object properties are updated every time the function is called. So, it can be used to show a conversion progress bar, alter some part of an application, or used to achieve any desired outcome.

The result in the code above is the outcome of the image recognition process. Each of the properties of result has the property bbox as the x/y coordinates of their bounding box.

Here are the properties of the result object, their meanings or uses: 

{   text: "I am codingnninja from Nigeria..."   hocr: "<div class='ocr_page' id= ..."   tsv: "1 1 0 0 0 0 0 0 1486 ..."   box: null   unlv: null   osd: null   confidence: 90   blocks: [{...}]   psm: "SINGLE_BLOCK"   oem: "DEFAULT"   version: "4.0.0-825-g887c"   paragraphs: [{...}]   lines: (5) [{...}, ...]   words: (47) [{...}, {...}, ...]   symbols: (240) [{...}, {...}, ...] }
  • text: All of the recognized text as a string.
  • lines: An array of every recognized line by line of text.
  • words: An array of every recognized word.
  • symbols: An array of each of the characters recognized.
  • paragraphs: An array of every recognized paragraph. We are going to discuss “confidence” later in this write-up.

Tesseract can also be used more imperatively as in:

import { createWorker } from 'tesseract.js';    const worker = createWorker({   logger: m => console.log(m)   });    (async () => {   await worker.load();   await worker.loadLanguage('eng');   await worker.initialize('eng');   const { data: { text } } = await     worker.recognize('https://tesseract.projectnaptha.com/img/eng_bw.png');  console.log(text);  await worker.terminate(); })();

This approach is related to the first approach but with different implementations.

createWorker(options) creates a web worker or node child process that creates a Tesseract worker. The worker helps set up the Tesseract OCR engine. The load() method loads the Tesseract core-scripts, loadLanguage() loads any language supplied to it as a string, initialize() makes sure Tesseract is fully ready for use and then the recognize method is used to process the image provided. The terminate() method stops the worker and cleans up everything.

Note: Please check Tesseract APIs documentation for more information.

Now, we have to build something to really see how effective Tesseract.js is.

What Are We Going To Build?

We are going to build a gift card PIN extractor because extracting PIN from a gift card was the issue that led to this writing adventure in the first place.

We will build a simple application that extracts the PIN from a scanned gift card. As I set out to build a simple gift card pin extractor, I will walk you through some of the challenges I faced along the line, the solutions I provided, and my conclusion based on my experience.

Below is the image we are going to use for testing because it has some realistic properties that are possible in the real world.

We will extract AQUX-QWMB6L-R6JAU from the card. So, let’s get started.

Installation Of React And Tesseract

There is a question to attend to before installing React and Tesseract.js and the question is, why using React with Tesseract? Practically, we can use Tesseract with Vanilla JavaScript, any JavaScript libraries or frameworks such React, Vue and Angular.

Using React in this case is a personal preference. Initially, I wanted to use Vue but I decided to go with React because I am more familiar with React than Vue.

Now, let’s continue with the installations.

To install React with create-react-app, you have to run the code below:

npx create-react-app image-to-text cd image-to-text yarn add Tesseract.js

or

npm install tesseract.js

I decided to go with yarn to install Tesseract.js because I was unable to install Tesseract with npm but yarn got the job done without stress. You can use npm but I recommend installing Tesseract with yarn judging from my experience.

Now, let’s start our development server by running the code below:

yarn start

or 

npm start

After running yarn start or npm start, your default browser should open a webpage that looks like below:

You could also navigate to localhost:3000 in the browser provided the page is not launched automatically.

After installing React and Tesseract.js, what next?

Setting Up An Upload Form

In this case, we are going to adjust the home page (App.js) we just viewed in the browser to contain the form we need:

import { useState, useRef } from 'react'; import Tesseract from 'tesseract.js'; import './App.css';  function App() {   const [imagePath, setImagePath] = useState("");   const [text, setText] = useState("");    const handleChange = (event) => {     setImage(URL.createObjectURL(event.target.files[0]));   }    return (     <div className="App">       <main className="App-main">         <h3>Actual image uploaded</h3>         <img             src={imagePath} className="App-logo" alt="logo"/>            <h3>Extracted text</h3>         <div className="text-box">           <p> {text} </p>         </div>         <input type="file" onChange={handleChange} />       </main>     </div>   ); }  export default App

The part of the code above that needs our attention at this point is the function handleChange.

const handleChange = (event) => {     setImage(URL.createObjectURL(event.target.files[0]));   }

In the function, URL.createObjectURL takes a selected file through event.target.files[0] and creates a reference URL that can be used with HTML tags such as img, audio and video. We used setImagePath to add the URL to the state. Now, the URL can now be accessed with imagePath.

<img src={imagePath} className="App-logo" alt="image"/>

We set the image’s src attribute to {imagePath} to preview it in the browser before processing it.

Converting Selected Images To Texts

As we have grabbed the path to the image selected, we can pass the image’s path to Tesseract.js to extract texts from it.

 import { useState} from 'react'; import Tesseract from 'tesseract.js'; import './App.css';  function App() {   const [imagePath, setImagePath] = useState("");   const [text, setText] = useState("");    const handleChange = (event) => {     setImagePath(URL.createObjectURL(event.target.files[0]));   }    const handleClick = () => {      Tesseract.recognize(       imagePath,'eng',       {          logger: m => console.log(m)        }     )     .catch (err => {       console.error(err);     })     .then(result => {       // Get Confidence score       let confidence = result.confidence        let text = result.text       setText(text);      })   }    return (     <div className="App">       <main className="App-main">         <h3>Actual imagePath uploaded</h3>         <img             src={imagePath} className="App-image" alt="logo"/>            <h3>Extracted text</h3>         <div className="text-box">           <p> {text} </p>         </div>         <input type="file" onChange={handleChange} />         <button onClick={handleClick} style={{height:50}}> convert to text</button>       </main>     </div>   ); }  export default App

We add the function “handleClick” to “App.js and it contains Tesseract.js API that takes the path to the selected image. Tesseract.js takes “imagePath”, “language”, “a setting object”.

The button below is added to the form to call “handClick” which triggers image-to-text conversion whenever the button is clicked.

<button onClick={handleClick} style={{height:50}}> convert to text</button>

When the processing is successful, we access both “confidence” and “text” from the result. Then, we add “text” to the state with “setText(text)”.

By adding to <p> {text} </p>, we display the extracted text.

It is obvious that “text” is extracted from the image but what is confidence?

Confidence shows how accurate the conversion is. The confidence level is between 1 to 100. 1 stands for the worst while 100 stands for the best in terms of accuracy. It can also be used to determine whether an extracted text should be accepted as accurate or not.

Then the question is what factors can affect the confidence score or the accuracy of the entire conversion? It is mostly affected by three major factors — the quality and nature of the document used, the quality of the scan created from the document and the processing abilities of the Tesseract engine.

Now, let’s add the code below to “App.css” to style the application a bit.

.App {   text-align: center; }  .App-image {   width: 60vmin;   pointer-events: none; }  .App-main {   background-color: #282c34;   min-height: 100vh;   display: flex;   flex-direction: column;   align-items: center;   justify-content: center;   font-size: calc(7px + 2vmin);   color: white; }  .text-box {   background: #fff;   color: #333;   border-radius: 5px;   text-align: center; }

Here is the result of my first test:

Outcome In Firefox

The confidence level of the result above is 64. It is worth noting that the gift card image is dark in color and it definitely affects the result we get.

If you take a closer look at the image above, you will see the pin from the card is almost accurate in the extracted text. It is not accurate because the gift card is not really clear.

Oh, wait! What will it look like in Chrome?

Outcome In Chrome

Ah! The outcome is even worse in Chrome. But why is the outcome in Chrome different from Mozilla Firefox? Different browsers handle images and their colour profiles differently. That means, an image can be rendered differently depending on the browser. By supplying pre-rendered image.data to Tesseract, it is likely to produce a different outcome in different browsers because different image.data is supplied to Tesseract depending on the browser in use. Preprocessing an image, as we will see later in this article, will help achieve a consistent result.

We need to be more accurate so that we can be sure we are getting or giving the right information. So we have to take it a bit further.

Let’s try more to see if we can achieve the aim in the end.

Testing For Accuracy

There are a lot of factors that affect an image-to-text conversion with Tesseract.js. Most of these factors revolve around the nature of the image we want to process and the rest depends on how the Tesseract engine handles the conversion.

Internally, Tesseract preprocesses images before the actual OCR conversion but it doesn’t always give accurate results.

As a solution, we can preprocess images to achieve accurate conversions. We can binarise, invert, dilate, deskew or rescale an image to preprocess it for Tesseract.js.

Image pre-processing is a lot of work or an extensive field on its own. Fortunately, P5.js has provided all the image preprocessing techniques we want to use. Instead of reinventing the wheel or using the whole of the library just because we want to use a tiny part of it, I have copied the ones we need. All the image preprocessing techniques are included in preprocess.js.

What Is Binarization?

Binarization is the conversion of the pixels of an image to either black or white. We want to binarize the previous gift card to check whether the accuracy will be better or not.

Previously, we extracted some texts from a gift card but the target PIN was not as accurate as we wanted. So there is a need to find another way to get an accurate result.

Now, we want to binarize the gift card, i.e. we want to convert its pixels to black and white so that we can see whether a better level of accuracy can be achieved or not.

The functions below will be used for binarization and it is included in a separate file called preprocess.js.

function preprocessImage(canvas) {     const ctx = canvas.getContext('2d');     const image = ctx.getImageData(0,0,canvas.width, canvas.height);     thresholdFilter(image.data, 0.5);     return image;  }   Export default preprocessImage

What does the code above do?

We introduce canvas to hold an image data to apply some filters, to pre-process the image, before passing it to Tesseract for conversion.

The first preprocessImage function is located in preprocess.js and prepares the canvas for use by getting its pixels. The function thresholdFilter binarizes the image by converting its pixels to either black or white.

Let’s call preprocessImage to see if the text extracted from the previous gift card can be more accurate.

By the time we update App.js, it should now look like the code this:

import { useState, useRef } from 'react'; import preprocessImage from './preprocess'; import Tesseract from 'tesseract.js'; import './App.css';  function App() {   const [image, setImage] = useState("");   const [text, setText] = useState("");   const canvasRef = useRef(null);   const imageRef = useRef(null);    const handleChange = (event) => {     setImage(URL.createObjectURL(event.target.files[0]))   }    const handleClick = () => {      const canvas = canvasRef.current;     const ctx = canvas.getContext('2d');      ctx.drawImage(imageRef.current, 0, 0);     ctx.putImageData(preprocessImage(canvas),0,0);     const dataUrl = canvas.toDataURL("image/jpeg");      Tesseract.recognize(       dataUrl,'eng',       {          logger: m => console.log(m)        }     )     .catch (err => {       console.error(err);     })     .then(result => {       // Get Confidence score       let confidence = result.confidence       console.log(confidence)       // Get full output       let text = result.text        setText(text);     })   }    return (     <div className="App">       <main className="App-main">         <h3>Actual image uploaded</h3>         <img             src={image} className="App-logo" alt="logo"            ref={imageRef}             />         <h3>Canvas</h3>         <canvas ref={canvasRef} width={700} height={250}></canvas>           <h3>Extracted text</h3>         <div className="pin-box">           <p> {text} </p>         </div>         <input type="file" onChange={handleChange} />         <button onClick={handleClick} style={{height:50}}>Convert to text</button>       </main>     </div>   ); }  export default App

First, we have to import “preprocessImage” from “preprocess.js” with the code below:

import preprocessImage from './preprocess';

Then, we add a canvas tag to the form. We set the ref attribute of both the canvas and the img tags to { canvasRef } and { imageRef } respectively. The refs are used to access the canvas and the image from the App component. We get hold of both the canvas and the image with “useRef” as in:

const canvasRef = useRef(null); const imageRef = useRef(null);

In this part of the code, we merge the image to the canvas as we can only preprocess a canvas in JavaScript. We then convert it to a data URL with “jpeg” as its image format.

const canvas = canvasRef.current; const ctx = canvas.getContext('2d');  ctx.drawImage(imageRef.current, 0, 0); ctx.putImageData(preprocessImage(canvas),0,0); const dataUrl = canvas.toDataURL("image/jpeg");

“dataUrl” is passed to Tesseract as the image to be processed.

Now, let’s check whether the text extracted will be more accurate.

Test #2

The image above shows the result in Firefox. It is obvious that the dark part of the image has been changed to white but preprocessing the image doesn’t lead to a more accurate result. It is even worse.

The first conversion only has two incorrect characters but this one has four incorrect characters. I even tried changing the threshold level but to no avail. We don’t get a better result not because binarization is bad but because binarizing the image doesn’t fix the nature of the image in a way that is suitable for the Tesseract engine.

Let’s check what it also looks like in Chrome:

We get the same outcome.

After getting a worse result by binarizing the image, there is a need to check other image preprocessing techniques to see whether we can solve the problem or not. So, we are going to try dilation, inversion, and blurring next.

Let’s just get the code for each of the techniques from P5.js as used by this article. We will add the image processing techniques to preprocess.js and use them one by one. It is necessary to understand each of the image preprocessing techniques we want to use before using them, so we are going to discuss them first.

What Is Dilation?

Dilation is adding pixels to the boundaries of objects in an image to make it wider, larger, or more open. The “dilate” technique is used to preprocess our images to increase the brightness of the objects on the images. We need a function to dilate images using JavaScript, so the code snippet to dilate an image is added to preprocess.js.

What Is Blur?

Blurring is smoothing the colors of an image by reducing its sharpness. Sometimes, images have small dots/patches. To remove those patches, we can blur the images. The code snippet to blur an image is included in preprocess.js.

What Is Inversion?

Inversion is changing light areas of an image to a dark color and dark areas to a light color. For example, if an image has a black background and white foreground, we can invert it so that its background will be white and its foreground will be black. We have also added the code snippet to invert an image to preprocess.js.

After adding dilate, invertColors and blurARGB to “preprocess.js”, we can now use them to preprocess images. To use them, we need to update the initial “preprocessImage” function in preprocess.js:

preprocessImage(...) now looks like this:

function preprocessImage(canvas) {   const level = 0.4;   const radius = 1;   const ctx = canvas.getContext('2d');   const image = ctx.getImageData(0,0,canvas.width, canvas.height);   blurARGB(image.data, canvas, radius);   dilate(image.data, canvas);   invertColors(image.data);   thresholdFilter(image.data, level);   return image;  }

In preprocessImage above, we apply four preprocessing techniques to an image: blurARGB() to remove the dots on the image, dilate() to increase the brightness of the image, invertColors() to switch the foreground and background color of the image and thresholdFilter() to convert the image to black and white which is more suitable for Tesseract conversion.

The thresholdFilter() takes image.data and level as its parameters. level is used to set how white or black the image should be. We determined the thresholdFilter level and blurRGB radius by trial and error as we are not sure how white, dark or smooth the image should be for Tesseract to produce a great result.

Test #3

Here is the new result after applying four techniques:

The image above represents the result we get in both Chrome and Firefox.

Oops! The outcome is terrible.

Instead of using all four techniques, why don’t we just use two of them at a time?

Yeah! We can simply use invertColors and thresholdFilter techniques to convert the image to black and white, and switch the foreground and the background of the image. But how do we know what and what techniques to combine? We know what to combine based on the nature of the image we want to preprocess.

For example, a digital image has to be converted to black and white, and an image with patches has to be blurred to remove the dots/patches. What really matters is to understand what each of the techniques is used for.

To use invertColors and thresholdFilter, we need to comment out both blurARGB and dilate in preprocessImage:

function preprocessImage(canvas) {     const ctx = canvas.getContext('2d');     const image = ctx.getImageData(0,0,canvas.width, canvas.height);     // blurARGB(image.data, canvas, 1);     // dilate(image.data, canvas);     invertColors(image.data);     thresholdFilter(image.data, 0.5);     return image; }

Test #4

Now, here is the new outcome:

The result is still worse than the one without any preprocessing. After adjusting each of the techniques for this particular image and some other images, I have come to the conclusion that images with different nature require different preprocessing techniques.

In short, using Tesseract.js without image preprocessing produced the best outcome for the gift card above. All other experiments with image preprocessing yielded less accurate outcomes.

Issue

Initially, I wanted to extract the PIN from any Amazon gift card but I couldn’t achieve that because there is no point to match an inconsistent PIN to get a consistent result. Although it is possible to process an image to get an accurate PIN, yet such preprocessing will be inconsistent by the time another image with different nature is used.

The Best Outcome Produced

The image below showcases the best outcome produced by the experiments.

Test #5

The texts on the image and the ones extracted are totally the same. The conversion has 100% accuracy. I tried to reproduce the result but I was only able to reproduce it when using images with similar nature.

Observation And Lessons

  • Some images that are not preprocessed may give different outcomes in different browsers. This claim is evident in the first test. The outcome in Firefox is different from the one in Chrome. However, preprocessing images helps achieve a consistent outcome in other tests.
  • Black color on a white background tends to give manageable results. The image below is an example of an accurate result without any preprocessing. I also was able to get the same level of accuracy by preprocessing the image but it took me a lot of adjustment which was unnecessary.

The conversion is 100% accurate.

  • A text with a big font size tends to be more accurate.

  • Fonts with curved edges tend to confuse Tesseract. The best result I got was achieved when I used Arial (font).
  • OCR is currently not good enough for automating image-to-text conversion, especially when more than 80% level of accuracy is required. However, it can be used to make the manual processing of texts on images less stressful by extracting texts for manual correction.
  • OCR is currently not good enough to pass useful information to screen readers for accessibility. Supplying inaccurate information to a screen reader can easily mislead or distract users.
  • OCR is very promising as neural networks make it possible to learn and improve. Deep learning will make OCR a game-changer in the near future.
  • Making decisions with confidence. A confidence score can be used to make decisions that can greatly impact our applications. The confidence score can be used to determine whether to accept or reject a result. From my experience and experiment, I realized that any confidence score below 90 isn’t really useful. If I only need to extract some pins from a text, I will expect a confidence score between 75 and 100, and anything below 75 will be rejected.

In case I am dealing with texts without the need to extract any part of it, I will definitely accept a confidence score between 90 to 100 but reject any score below that. For example, 90 and above accuracy will be expected if I want to digitize documents such as cheques, a historic draft or whenever an exact copy is necessary. But a score that is between 75 and 90 is acceptable when an exact copy is not important such as getting the PIN from a gift card. In short, a confidence score helps in making decisions that impact our applications.

Conclusion

Given the data processing limitation caused by texts on images and the disadvantages associated with it, Optical Character Recognition (OCR) is a useful technology to embrace. Although OCR has its limitations, it is very promising because of its use of neural networks.

Over time, OCR will overcome most of its limitations with the help of deep learning, but before then, the approaches highlighted in this article can be utilized to deal with text extraction from images, at least, to reduce the hardship and losses associated with manual processing — especially from a business point of view.

It is now your turn to try OCR to extract texts from images. Good luck!

Further Reading

Articles on Smashing Magazine — For Web Designers And Developers

21 Jun

Image To Text Conversion With React And Tesseract.js (OCR)

by TBSCategories: News

Data is the backbone of every software application because the main purpose of an application is to solve human problems. To solve human problems, it is necessary to have some information about them.

Such information is represented as data, especially through computation. On the web, data is mostly collected in the form of texts, images, videos, and many more. Sometimes, images contain essential texts that are meant to be processed to achieve a certain purpose. These images were mostly processed manually because there was no way to process them programmatically.

The inability to extract text from images was a data processing limitation I experienced first-hand at my last company. We needed to process scanned gift cards and we had to do it manually since we couldn’t extract text from images.

There was a department called “Operations” within the company that was responsible for manual confirming gift cards and crediting users’ accounts. Although we had a website through which users connected with us, the processing of gift cards was carried out manually behind the scenes.

At the time, our website was built mainly with PHP (Laravel) for the backend and JavaScript (jQuery and Vue) for the frontend. Our technical stack was good enough to work with Tesseract.js provided the issue was considered important by the management.

I was willing to solve the problem but it was not necessary to solve the problem judging from the business’ or the management’s point of view. After leaving the company, I decided to do some research and try to find possible solutions. Eventually, I discovered OCR.

What Is OCR?

OCR stands for “Optical Character Recognition” or “Optical Character Reader”. It is used to extract texts from images.

The Evolution Of OCR can be traced to several inventions but Optophone, “Gismo” , CCD flatbed scanner, Newton MesssagePad and Tesseract are the major inventions that take character recognition to another level of usefulness.

So, why use OCR? Well, Optical Character Recognition solves a lot of problems, one of which triggered me to write this article. I realized the ability to extract texts from an image ensures a lot of possibilities such as:

  • Regulation
    Every organization needs to regulate users’ activities for some reasons. The regulation might be used to protect users’ rights and secure them from threats or scams.
    Extracting texts from an image enables an organization to process textual information on an image for regulation, especially when the images are supplied by some of the users.
    For example, Facebook-like regulation of the number of texts on images used for ads can be achieved with OCR. Also, hiding sensitive content on Twitter is also made possible by OCR.
  • Searchability
    Searching is one of the most common activities, especially on the internet. Searching algorithms are mostly based on manipulating texts. With Optical Character Recognition, it is possible to recognize characters on images and use them to provide relevant image results to users. In short, images and videos are now searchable with the aid of OCR.
  • Accessibility
    Having texts on images has always been a challenge for accessibility and it is the rule of thumb to have few texts on an image. With OCR, screen readers can have access to texts on images to provide some necessary experience to its users.
  • Data Processing Automation The processing of data is mostly automated for scale. Having texts on images is a limitation to data processing because the texts cannot be processed except manually. Optical Character Recognition (OCR) makes it possible to extract texts on images programmatically thereby, ensuring data processing automation especially when it has to do with the processing of texts on images.
  • Digitization Of Printed Materials
    Everything is going digital and there are still a lot of documents to be digitized. Cheques, certificates, and other physical documents can now be digitized with the use of Optical Character Recognition.

Finding out all the uses above deepened my interests, so I decided to go further by asking a question:

“How can I use OCR on the web, especially in a React application?”

That question led me to Tesseract.js.

What Is Tesseract.js?

[Tesseract.js is a JavaScript library that compiles the original Tesseract from C to JavaScript WebAssembly thereby making OCR accessible in the browser. Tesseract.js engine was originally written in ASM.js and it was later ported to WebAssembly but ASM.js still serves as a backup in some cases when WebAssembly is not supported.

As stated on the website of Tesseract.js, it supports more than 100 languages, automatic text orientation and script detection, a simple interface for reading paragraphs, words and character bounding boxes.

Tesseract is an optical character recognition engine for various operating systems. It is free software, released under the Apache Licence. Hewlett-Packard developed Tesseract as proprietary software in the 1980s. It was released as open source in 2005 and its development has been sponsored by Google since 2006.

The latest version, version 4, of Tesseract was released in October 2018 and it contains a new OCR engine that uses a neural network system based on Long Short-Term Memory (LSTM) and it is meant to produce more accurate results.

Understanding Tesseract APIs

To really understand how Tesseract works, we need to break down some of its APIs and their components. According to the Tesseract.js documentation, there are two ways to approach using it. Below is the first approach an its break down:

Tesseract.recognize(   image,language,   {      logger: m => console.log(m)    } ) .catch (err => {   console.error(err); }) .then(result => {  console.log(result); }) }

The recognize method takes image as its first argument, language (which can be multiple) as its second argument and { logger: m => console.log(me) } as its last argument. The image format supported by Tesseract are jpg, png, bmp and pbm which can only be supplied as elements (img, video or canvas), file object (<input>), blob object, path or URL to an image and base64 encoded image. (Read here for more information about all of the image formats Tesseract can handle.)

Language is supplied as a string such as eng. The + sign could be used to concatenate several languages as in eng+chi_tra. The language argument is used to determine the trained language data to be used in processing of images.

Note: You’ll find all of the available languages and their codes over here.

{ logger: m => console.log(m) } is very useful to get information about the progress of an image being processed. The logger property takes a function that will be called multiple times as Tesseract processes an image. The parameter to the logger function should be an object with workerId, jobId, status and progress as its properties: 

{ workerId: ‘worker-200030’, jobId: ‘job-734747’, status: ‘recognizing text’, progress: ‘0.9’ }

progress is a number between 0 and 1, and it is in percentage to show the progress of an image recognition process.

Tesseract automatically generates the object as a parameter to the logger function but it can also be supplied manually. As a recognition process is taking place, the logger object properties are updated every time the function is called. So, it can be used to show a conversion progress bar, alter some part of an application, or used to achieve any desired outcome.

The result in the code above is the outcome of the image recognition process. Each of the properties of result has the property bbox as the x/y coordinates of their bounding box.

Here are the properties of the result object, their meanings or uses: 

{   text: "I am codingnninja from Nigeria..."   hocr: "<div class='ocr_page' id= ..."   tsv: "1 1 0 0 0 0 0 0 1486 ..."   box: null   unlv: null   osd: null   confidence: 90   blocks: [{...}]   psm: "SINGLE_BLOCK"   oem: "DEFAULT"   version: "4.0.0-825-g887c"   paragraphs: [{...}]   lines: (5) [{...}, ...]   words: (47) [{...}, {...}, ...]   symbols: (240) [{...}, {...}, ...] }
  • text: All of the recognized text as a string.
  • lines: An array of every recognized line by line of text.
  • words: An array of every recognized word.
  • symbols: An array of each of the characters recognized.
  • paragraphs: An array of every recognized paragraph. We are going to discuss “confidence” later in this write-up.

Tesseract can also be used more imperatively as in:

import { createWorker } from 'tesseract.js';    const worker = createWorker({   logger: m => console.log(m)   });    (async () => {   await worker.load();   await worker.loadLanguage('eng');   await worker.initialize('eng');   const { data: { text } } = await     worker.recognize('https://tesseract.projectnaptha.com/img/eng_bw.png');  console.log(text);  await worker.terminate(); })();

This approach is related to the first approach but with different implementations.

createWorker(options) creates a web worker or node child process that creates a Tesseract worker. The worker helps set up the Tesseract OCR engine. The load() method loads the Tesseract core-scripts, loadLanguage() loads any language supplied to it as a string, initialize() makes sure Tesseract is fully ready for use and then the recognize method is used to process the image provided. The terminate() method stops the worker and cleans up everything.

Note: Please check Tesseract APIs documentation for more information.

Now, we have to build something to really see how effective Tesseract.js is.

What Are We Going To Build?

We are going to build a gift card PIN extractor because extracting PIN from a gift card was the issue that led to this writing adventure in the first place.

We will build a simple application that extracts the PIN from a scanned gift card. As I set out to build a simple gift card pin extractor, I will walk you through some of the challenges I faced along the line, the solutions I provided, and my conclusion based on my experience.

Below is the image we are going to use for testing because it has some realistic properties that are possible in the real world.

We will extract AQUX-QWMB6L-R6JAU from the card. So, let’s get started.

Installation Of React And Tesseract

There is a question to attend to before installing React and Tesseract.js and the question is, why using React with Tesseract? Practically, we can use Tesseract with Vanilla JavaScript, any JavaScript libraries or frameworks such React, Vue and Angular.

Using React in this case is a personal preference. Initially, I wanted to use Vue but I decided to go with React because I am more familiar with React than Vue.

Now, let’s continue with the installations.

To install React with create-react-app, you have to run the code below:

npx create-react-app image-to-text cd image-to-text yarn add Tesseract.js

or

npm install tesseract.js

I decided to go with yarn to install Tesseract.js because I was unable to install Tesseract with npm but yarn got the job done without stress. You can use npm but I recommend installing Tesseract with yarn judging from my experience.

Now, let’s start our development server by running the code below:

yarn start

or 

npm start

After running yarn start or npm start, your default browser should open a webpage that looks like below:

You could also navigate to localhost:3000 in the browser provided the page is not launched automatically.

After installing React and Tesseract.js, what next?

Setting Up An Upload Form

In this case, we are going to adjust the home page (App.js) we just viewed in the browser to contain the form we need:

import { useState, useRef } from 'react'; import Tesseract from 'tesseract.js'; import './App.css';  function App() {   const [imagePath, setImagePath] = useState("");   const [text, setText] = useState("");    const handleChange = (event) => {     setImage(URL.createObjectURL(event.target.files[0]));   }    return (     <div className="App">       <main className="App-main">         <h3>Actual image uploaded</h3>         <img             src={imagePath} className="App-logo" alt="logo"/>            <h3>Extracted text</h3>         <div className="text-box">           <p> {text} </p>         </div>         <input type="file" onChange={handleChange} />       </main>     </div>   ); }  export default App

The part of the code above that needs our attention at this point is the function handleChange.

const handleChange = (event) => {     setImage(URL.createObjectURL(event.target.files[0]));   }

In the function, URL.createObjectURL takes a selected file through event.target.files[0] and creates a reference URL that can be used with HTML tags such as img, audio and video. We used setImagePath to add the URL to the state. Now, the URL can now be accessed with imagePath.

<img src={imagePath} className="App-logo" alt="image"/>

We set the image’s src attribute to {imagePath} to preview it in the browser before processing it.

Converting Selected Images To Texts

As we have grabbed the path to the image selected, we can pass the image’s path to Tesseract.js to extract texts from it.

 import { useState} from 'react'; import Tesseract from 'tesseract.js'; import './App.css';  function App() {   const [imagePath, setImagePath] = useState("");   const [text, setText] = useState("");    const handleChange = (event) => {     setImagePath(URL.createObjectURL(event.target.files[0]));   }    const handleClick = () => {      Tesseract.recognize(       imagePath,'eng',       {          logger: m => console.log(m)        }     )     .catch (err => {       console.error(err);     })     .then(result => {       // Get Confidence score       let confidence = result.confidence        let text = result.text       setText(text);      })   }    return (     <div className="App">       <main className="App-main">         <h3>Actual imagePath uploaded</h3>         <img             src={imagePath} className="App-image" alt="logo"/>            <h3>Extracted text</h3>         <div className="text-box">           <p> {text} </p>         </div>         <input type="file" onChange={handleChange} />         <button onClick={handleClick} style={{height:50}}> convert to text</button>       </main>     </div>   ); }  export default App

We add the function “handleClick” to “App.js and it contains Tesseract.js API that takes the path to the selected image. Tesseract.js takes “imagePath”, “language”, “a setting object”.

The button below is added to the form to call “handClick” which triggers image-to-text conversion whenever the button is clicked.

<button onClick={handleClick} style={{height:50}}> convert to text</button>

When the processing is successful, we access both “confidence” and “text” from the result. Then, we add “text” to the state with “setText(text)”.

By adding to <p> {text} </p>, we display the extracted text.

It is obvious that “text” is extracted from the image but what is confidence?

Confidence shows how accurate the conversion is. The confidence level is between 1 to 100. 1 stands for the worst while 100 stands for the best in terms of accuracy. It can also be used to determine whether an extracted text should be accepted as accurate or not.

Then the question is what factors can affect the confidence score or the accuracy of the entire conversion? It is mostly affected by three major factors — the quality and nature of the document used, the quality of the scan created from the document and the processing abilities of the Tesseract engine.

Now, let’s add the code below to “App.css” to style the application a bit.

.App {   text-align: center; }  .App-image {   width: 60vmin;   pointer-events: none; }  .App-main {   background-color: #282c34;   min-height: 100vh;   display: flex;   flex-direction: column;   align-items: center;   justify-content: center;   font-size: calc(7px + 2vmin);   color: white; }  .text-box {   background: #fff;   color: #333;   border-radius: 5px;   text-align: center; }

Here is the result of my first test:

Outcome In Firefox

The confidence level of the result above is 64. It is worth noting that the gift card image is dark in color and it definitely affects the result we get.

If you take a closer look at the image above, you will see the pin from the card is almost accurate in the extracted text. It is not accurate because the gift card is not really clear.

Oh, wait! What will it look like in Chrome?

Outcome In Chrome

Ah! The outcome is even worse in Chrome. But why is the outcome in Chrome different from Mozilla Firefox? Different browsers handle images and their colour profiles differently. That means, an image can be rendered differently depending on the browser. By supplying pre-rendered image.data to Tesseract, it is likely to produce a different outcome in different browsers because different image.data is supplied to Tesseract depending on the browser in use. Preprocessing an image, as we will see later in this article, will help achieve a consistent result.

We need to be more accurate so that we can be sure we are getting or giving the right information. So we have to take it a bit further.

Let’s try more to see if we can achieve the aim in the end.

Testing For Accuracy

There are a lot of factors that affect an image-to-text conversion with Tesseract.js. Most of these factors revolve around the nature of the image we want to process and the rest depends on how the Tesseract engine handles the conversion.

Internally, Tesseract preprocesses images before the actual OCR conversion but it doesn’t always give accurate results.

As a solution, we can preprocess images to achieve accurate conversions. We can binarise, invert, dilate, deskew or rescale an image to preprocess it for Tesseract.js.

Image pre-processing is a lot of work or an extensive field on its own. Fortunately, P5.js has provided all the image preprocessing techniques we want to use. Instead of reinventing the wheel or using the whole of the library just because we want to use a tiny part of it, I have copied the ones we need. All the image preprocessing techniques are included in preprocess.js.

What Is Binarization?

Binarization is the conversion of the pixels of an image to either black or white. We want to binarize the previous gift card to check whether the accuracy will be better or not.

Previously, we extracted some texts from a gift card but the target PIN was not as accurate as we wanted. So there is a need to find another way to get an accurate result.

Now, we want to binarize the gift card, i.e. we want to convert its pixels to black and white so that we can see whether a better level of accuracy can be achieved or not.

The functions below will be used for binarization and it is included in a separate file called preprocess.js.

function preprocessImage(canvas) {     const ctx = canvas.getContext('2d');     const image = ctx.getImageData(0,0,canvas.width, canvas.height);     thresholdFilter(image.data, 0.5);     return image;  }   Export default preprocessImage

What does the code above do?

We introduce canvas to hold an image data to apply some filters, to pre-process the image, before passing it to Tesseract for conversion.

The first preprocessImage function is located in preprocess.js and prepares the canvas for use by getting its pixels. The function thresholdFilter binarizes the image by converting its pixels to either black or white.

Let’s call preprocessImage to see if the text extracted from the previous gift card can be more accurate.

By the time we update App.js, it should now look like the code this:

import { useState, useRef } from 'react'; import preprocessImage from './preprocess'; import Tesseract from 'tesseract.js'; import './App.css';  function App() {   const [image, setImage] = useState("");   const [text, setText] = useState("");   const canvasRef = useRef(null);   const imageRef = useRef(null);    const handleChange = (event) => {     setImage(URL.createObjectURL(event.target.files[0]))   }    const handleClick = () => {      const canvas = canvasRef.current;     const ctx = canvas.getContext('2d');      ctx.drawImage(imageRef.current, 0, 0);     ctx.putImageData(preprocessImage(canvas),0,0);     const dataUrl = canvas.toDataURL("image/jpeg");      Tesseract.recognize(       dataUrl,'eng',       {          logger: m => console.log(m)        }     )     .catch (err => {       console.error(err);     })     .then(result => {       // Get Confidence score       let confidence = result.confidence       console.log(confidence)       // Get full output       let text = result.text        setText(text);     })   }    return (     <div className="App">       <main className="App-main">         <h3>Actual image uploaded</h3>         <img             src={image} className="App-logo" alt="logo"            ref={imageRef}             />         <h3>Canvas</h3>         <canvas ref={canvasRef} width={700} height={250}></canvas>           <h3>Extracted text</h3>         <div className="pin-box">           <p> {text} </p>         </div>         <input type="file" onChange={handleChange} />         <button onClick={handleClick} style={{height:50}}>Convert to text</button>       </main>     </div>   ); }  export default App

First, we have to import “preprocessImage” from “preprocess.js” with the code below:

import preprocessImage from './preprocess';

Then, we add a canvas tag to the form. We set the ref attribute of both the canvas and the img tags to { canvasRef } and { imageRef } respectively. The refs are used to access the canvas and the image from the App component. We get hold of both the canvas and the image with “useRef” as in:

const canvasRef = useRef(null); const imageRef = useRef(null);

In this part of the code, we merge the image to the canvas as we can only preprocess a canvas in JavaScript. We then convert it to a data URL with “jpeg” as its image format.

const canvas = canvasRef.current; const ctx = canvas.getContext('2d');  ctx.drawImage(imageRef.current, 0, 0); ctx.putImageData(preprocessImage(canvas),0,0); const dataUrl = canvas.toDataURL("image/jpeg");

“dataUrl” is passed to Tesseract as the image to be processed.

Now, let’s check whether the text extracted will be more accurate.

Test #2

The image above shows the result in Firefox. It is obvious that the dark part of the image has been changed to white but preprocessing the image doesn’t lead to a more accurate result. It is even worse.

The first conversion only has two incorrect characters but this one has four incorrect characters. I even tried changing the threshold level but to no avail. We don’t get a better result not because binarization is bad but because binarizing the image doesn’t fix the nature of the image in a way that is suitable for the Tesseract engine.

Let’s check what it also looks like in Chrome:

We get the same outcome.

After getting a worse result by binarizing the image, there is a need to check other image preprocessing techniques to see whether we can solve the problem or not. So, we are going to try dilation, inversion, and blurring next.

Let’s just get the code for each of the techniques from P5.js as used by this article. We will add the image processing techniques to preprocess.js and use them one by one. It is necessary to understand each of the image preprocessing techniques we want to use before using them, so we are going to discuss them first.

What Is Dilation?

Dilation is adding pixels to the boundaries of objects in an image to make it wider, larger, or more open. The “dilate” technique is used to preprocess our images to increase the brightness of the objects on the images. We need a function to dilate images using JavaScript, so the code snippet to dilate an image is added to preprocess.js.

What Is Blur?

Blurring is smoothing the colors of an image by reducing its sharpness. Sometimes, images have small dots/patches. To remove those patches, we can blur the images. The code snippet to blur an image is included in preprocess.js.

What Is Inversion?

Inversion is changing light areas of an image to a dark color and dark areas to a light color. For example, if an image has a black background and white foreground, we can invert it so that its background will be white and its foreground will be black. We have also added the code snippet to invert an image to preprocess.js.

After adding dilate, invertColors and blurARGB to “preprocess.js”, we can now use them to preprocess images. To use them, we need to update the initial “preprocessImage” function in preprocess.js:

preprocessImage(...) now looks like this:

function preprocessImage(canvas) {   const level = 0.4;   const radius = 1;   const ctx = canvas.getContext('2d');   const image = ctx.getImageData(0,0,canvas.width, canvas.height);   blurARGB(image.data, canvas, radius);   dilate(image.data, canvas);   invertColors(image.data);   thresholdFilter(image.data, level);   return image;  }

In preprocessImage above, we apply four preprocessing techniques to an image: blurARGB() to remove the dots on the image, dilate() to increase the brightness of the image, invertColors() to switch the foreground and background color of the image and thresholdFilter() to convert the image to black and white which is more suitable for Tesseract conversion.

The thresholdFilter() takes image.data and level as its parameters. level is used to set how white or black the image should be. We determined the thresholdFilter level and blurRGB radius by trial and error as we are not sure how white, dark or smooth the image should be for Tesseract to produce a great result.

Test #3

Here is the new result after applying four techniques:

The image above represents the result we get in both Chrome and Firefox.

Oops! The outcome is terrible.

Instead of using all four techniques, why don’t we just use two of them at a time?

Yeah! We can simply use invertColors and thresholdFilter techniques to convert the image to black and white, and switch the foreground and the background of the image. But how do we know what and what techniques to combine? We know what to combine based on the nature of the image we want to preprocess.

For example, a digital image has to be converted to black and white, and an image with patches has to be blurred to remove the dots/patches. What really matters is to understand what each of the techniques is used for.

To use invertColors and thresholdFilter, we need to comment out both blurARGB and dilate in preprocessImage:

function preprocessImage(canvas) {     const ctx = canvas.getContext('2d');     const image = ctx.getImageData(0,0,canvas.width, canvas.height);     // blurARGB(image.data, canvas, 1);     // dilate(image.data, canvas);     invertColors(image.data);     thresholdFilter(image.data, 0.5);     return image; }

Test #4

Now, here is the new outcome:

The result is still worse than the one without any preprocessing. After adjusting each of the techniques for this particular image and some other images, I have come to the conclusion that images with different nature require different preprocessing techniques.

In short, using Tesseract.js without image preprocessing produced the best outcome for the gift card above. All other experiments with image preprocessing yielded less accurate outcomes.

Issue

Initially, I wanted to extract the PIN from any Amazon gift card but I couldn’t achieve that because there is no point to match an inconsistent PIN to get a consistent result. Although it is possible to process an image to get an accurate PIN, yet such preprocessing will be inconsistent by the time another image with different nature is used.

The Best Outcome Produced

The image below showcases the best outcome produced by the experiments.

Test #5

The texts on the image and the ones extracted are totally the same. The conversion has 100% accuracy. I tried to reproduce the result but I was only able to reproduce it when using images with similar nature.

Observation And Lessons

  • Some images that are not preprocessed may give different outcomes in different browsers. This claim is evident in the first test. The outcome in Firefox is different from the one in Chrome. However, preprocessing images helps achieve a consistent outcome in other tests.
  • Black color on a white background tends to give manageable results. The image below is an example of an accurate result without any preprocessing. I also was able to get the same level of accuracy by preprocessing the image but it took me a lot of adjustment which was unnecessary.

The conversion is 100% accurate.

  • A text with a big font size tends to be more accurate.

  • Fonts with curved edges tend to confuse Tesseract. The best result I got was achieved when I used Arial (font).
  • OCR is currently not good enough for automating image-to-text conversion, especially when more than 80% level of accuracy is required. However, it can be used to make the manual processing of texts on images less stressful by extracting texts for manual correction.
  • OCR is currently not good enough to pass useful information to screen readers for accessibility. Supplying inaccurate information to a screen reader can easily mislead or distract users.
  • OCR is very promising as neural networks make it possible to learn and improve. Deep learning will make OCR a game-changer in the near future.
  • Making decisions with confidence. A confidence score can be used to make decisions that can greatly impact our applications. The confidence score can be used to determine whether to accept or reject a result. From my experience and experiment, I realized that any confidence score below 90 isn’t really useful. If I only need to extract some pins from a text, I will expect a confidence score between 75 and 100, and anything below 75 will be rejected.

In case I am dealing with texts without the need to extract any part of it, I will definitely accept a confidence score between 90 to 100 but reject any score below that. For example, 90 and above accuracy will be expected if I want to digitize documents such as cheques, a historic draft or whenever an exact copy is necessary. But a score that is between 75 and 90 is acceptable when an exact copy is not important such as getting the PIN from a gift card. In short, a confidence score helps in making decisions that impact our applications.

Conclusion

Given the data processing limitation caused by texts on images and the disadvantages associated with it, Optical Character Recognition (OCR) is a useful technology to embrace. Although OCR has its limitations, it is very promising because of its use of neural networks.

Over time, OCR will overcome most of its limitations with the help of deep learning, but before then, the approaches highlighted in this article can be utilized to deal with text extraction from images, at least, to reduce the hardship and losses associated with manual processing — especially from a business point of view.

It is now your turn to try OCR to extract texts from images. Good luck!

Further Reading

Articles on Smashing Magazine — For Web Designers And Developers

21 Jun

Image To Text Conversion With React And Tesseract.js (OCR)

by TBSCategories: News

Data is the backbone of every software application because the main purpose of an application is to solve human problems. To solve human problems, it is necessary to have some information about them.

Such information is represented as data, especially through computation. On the web, data is mostly collected in the form of texts, images, videos, and many more. Sometimes, images contain essential texts that are meant to be processed to achieve a certain purpose. These images were mostly processed manually because there was no way to process them programmatically.

The inability to extract text from images was a data processing limitation I experienced first-hand at my last company. We needed to process scanned gift cards and we had to do it manually since we couldn’t extract text from images.

There was a department called “Operations” within the company that was responsible for manual confirming gift cards and crediting users’ accounts. Although we had a website through which users connected with us, the processing of gift cards was carried out manually behind the scenes.

At the time, our website was built mainly with PHP (Laravel) for the backend and JavaScript (jQuery and Vue) for the frontend. Our technical stack was good enough to work with Tesseract.js provided the issue was considered important by the management.

I was willing to solve the problem but it was not necessary to solve the problem judging from the business’ or the management’s point of view. After leaving the company, I decided to do some research and try to find possible solutions. Eventually, I discovered OCR.

What Is OCR?

OCR stands for “Optical Character Recognition” or “Optical Character Reader”. It is used to extract texts from images.

The Evolution Of OCR can be traced to several inventions but Optophone, “Gismo” , CCD flatbed scanner, Newton MesssagePad and Tesseract are the major inventions that take character recognition to another level of usefulness.

So, why use OCR? Well, Optical Character Recognition solves a lot of problems, one of which triggered me to write this article. I realized the ability to extract texts from an image ensures a lot of possibilities such as:

  • Regulation
    Every organization needs to regulate users’ activities for some reasons. The regulation might be used to protect users’ rights and secure them from threats or scams.
    Extracting texts from an image enables an organization to process textual information on an image for regulation, especially when the images are supplied by some of the users.
    For example, Facebook-like regulation of the number of texts on images used for ads can be achieved with OCR. Also, hiding sensitive content on Twitter is also made possible by OCR.
  • Searchability
    Searching is one of the most common activities, especially on the internet. Searching algorithms are mostly based on manipulating texts. With Optical Character Recognition, it is possible to recognize characters on images and use them to provide relevant image results to users. In short, images and videos are now searchable with the aid of OCR.
  • Accessibility
    Having texts on images has always been a challenge for accessibility and it is the rule of thumb to have few texts on an image. With OCR, screen readers can have access to texts on images to provide some necessary experience to its users.
  • Data Processing Automation The processing of data is mostly automated for scale. Having texts on images is a limitation to data processing because the texts cannot be processed except manually. Optical Character Recognition (OCR) makes it possible to extract texts on images programmatically thereby, ensuring data processing automation especially when it has to do with the processing of texts on images.
  • Digitization Of Printed Materials
    Everything is going digital and there are still a lot of documents to be digitized. Cheques, certificates, and other physical documents can now be digitized with the use of Optical Character Recognition.

Finding out all the uses above deepened my interests, so I decided to go further by asking a question:

“How can I use OCR on the web, especially in a React application?”

That question led me to Tesseract.js.

What Is Tesseract.js?

[Tesseract.js is a JavaScript library that compiles the original Tesseract from C to JavaScript WebAssembly thereby making OCR accessible in the browser. Tesseract.js engine was originally written in ASM.js and it was later ported to WebAssembly but ASM.js still serves as a backup in some cases when WebAssembly is not supported.

As stated on the website of Tesseract.js, it supports more than 100 languages, automatic text orientation and script detection, a simple interface for reading paragraphs, words and character bounding boxes.

Tesseract is an optical character recognition engine for various operating systems. It is free software, released under the Apache Licence. Hewlett-Packard developed Tesseract as proprietary software in the 1980s. It was released as open source in 2005 and its development has been sponsored by Google since 2006.

The latest version, version 4, of Tesseract was released in October 2018 and it contains a new OCR engine that uses a neural network system based on Long Short-Term Memory (LSTM) and it is meant to produce more accurate results.

Understanding Tesseract APIs

To really understand how Tesseract works, we need to break down some of its APIs and their components. According to the Tesseract.js documentation, there are two ways to approach using it. Below is the first approach an its break down:

Tesseract.recognize(   image,language,   {      logger: m => console.log(m)    } ) .catch (err => {   console.error(err); }) .then(result => {  console.log(result); }) }

The recognize method takes image as its first argument, language (which can be multiple) as its second argument and { logger: m => console.log(me) } as its last argument. The image format supported by Tesseract are jpg, png, bmp and pbm which can only be supplied as elements (img, video or canvas), file object (<input>), blob object, path or URL to an image and base64 encoded image. (Read here for more information about all of the image formats Tesseract can handle.)

Language is supplied as a string such as eng. The + sign could be used to concatenate several languages as in eng+chi_tra. The language argument is used to determine the trained language data to be used in processing of images.

Note: You’ll find all of the available languages and their codes over here.

{ logger: m => console.log(m) } is very useful to get information about the progress of an image being processed. The logger property takes a function that will be called multiple times as Tesseract processes an image. The parameter to the logger function should be an object with workerId, jobId, status and progress as its properties: 

{ workerId: ‘worker-200030’, jobId: ‘job-734747’, status: ‘recognizing text’, progress: ‘0.9’ }

progress is a number between 0 and 1, and it is in percentage to show the progress of an image recognition process.

Tesseract automatically generates the object as a parameter to the logger function but it can also be supplied manually. As a recognition process is taking place, the logger object properties are updated every time the function is called. So, it can be used to show a conversion progress bar, alter some part of an application, or used to achieve any desired outcome.

The result in the code above is the outcome of the image recognition process. Each of the properties of result has the property bbox as the x/y coordinates of their bounding box.

Here are the properties of the result object, their meanings or uses: 

{   text: "I am codingnninja from Nigeria..."   hocr: "<div class='ocr_page' id= ..."   tsv: "1 1 0 0 0 0 0 0 1486 ..."   box: null   unlv: null   osd: null   confidence: 90   blocks: [{...}]   psm: "SINGLE_BLOCK"   oem: "DEFAULT"   version: "4.0.0-825-g887c"   paragraphs: [{...}]   lines: (5) [{...}, ...]   words: (47) [{...}, {...}, ...]   symbols: (240) [{...}, {...}, ...] }
  • text: All of the recognized text as a string.
  • lines: An array of every recognized line by line of text.
  • words: An array of every recognized word.
  • symbols: An array of each of the characters recognized.
  • paragraphs: An array of every recognized paragraph. We are going to discuss “confidence” later in this write-up.

Tesseract can also be used more imperatively as in:

import { createWorker } from 'tesseract.js';    const worker = createWorker({   logger: m => console.log(m)   });    (async () => {   await worker.load();   await worker.loadLanguage('eng');   await worker.initialize('eng');   const { data: { text } } = await     worker.recognize('https://tesseract.projectnaptha.com/img/eng_bw.png');  console.log(text);  await worker.terminate(); })();

This approach is related to the first approach but with different implementations.

createWorker(options) creates a web worker or node child process that creates a Tesseract worker. The worker helps set up the Tesseract OCR engine. The load() method loads the Tesseract core-scripts, loadLanguage() loads any language supplied to it as a string, initialize() makes sure Tesseract is fully ready for use and then the recognize method is used to process the image provided. The terminate() method stops the worker and cleans up everything.

Note: Please check Tesseract APIs documentation for more information.

Now, we have to build something to really see how effective Tesseract.js is.

What Are We Going To Build?

We are going to build a gift card PIN extractor because extracting PIN from a gift card was the issue that led to this writing adventure in the first place.

We will build a simple application that extracts the PIN from a scanned gift card. As I set out to build a simple gift card pin extractor, I will walk you through some of the challenges I faced along the line, the solutions I provided, and my conclusion based on my experience.

Below is the image we are going to use for testing because it has some realistic properties that are possible in the real world.

We will extract AQUX-QWMB6L-R6JAU from the card. So, let’s get started.

Installation Of React And Tesseract

There is a question to attend to before installing React and Tesseract.js and the question is, why using React with Tesseract? Practically, we can use Tesseract with Vanilla JavaScript, any JavaScript libraries or frameworks such React, Vue and Angular.

Using React in this case is a personal preference. Initially, I wanted to use Vue but I decided to go with React because I am more familiar with React than Vue.

Now, let’s continue with the installations.

To install React with create-react-app, you have to run the code below:

npx create-react-app image-to-text cd image-to-text yarn add Tesseract.js

or

npm install tesseract.js

I decided to go with yarn to install Tesseract.js because I was unable to install Tesseract with npm but yarn got the job done without stress. You can use npm but I recommend installing Tesseract with yarn judging from my experience.

Now, let’s start our development server by running the code below:

yarn start

or 

npm start

After running yarn start or npm start, your default browser should open a webpage that looks like below:

You could also navigate to localhost:3000 in the browser provided the page is not launched automatically.

After installing React and Tesseract.js, what next?

Setting Up An Upload Form

In this case, we are going to adjust the home page (App.js) we just viewed in the browser to contain the form we need:

import { useState, useRef } from 'react'; import Tesseract from 'tesseract.js'; import './App.css';  function App() {   const [imagePath, setImagePath] = useState("");   const [text, setText] = useState("");    const handleChange = (event) => {     setImage(URL.createObjectURL(event.target.files[0]));   }    return (     <div className="App">       <main className="App-main">         <h3>Actual image uploaded</h3>         <img             src={imagePath} className="App-logo" alt="logo"/>            <h3>Extracted text</h3>         <div className="text-box">           <p> {text} </p>         </div>         <input type="file" onChange={handleChange} />       </main>     </div>   ); }  export default App

The part of the code above that needs our attention at this point is the function handleChange.

const handleChange = (event) => {     setImage(URL.createObjectURL(event.target.files[0]));   }

In the function, URL.createObjectURL takes a selected file through event.target.files[0] and creates a reference URL that can be used with HTML tags such as img, audio and video. We used setImagePath to add the URL to the state. Now, the URL can now be accessed with imagePath.

<img src={imagePath} className="App-logo" alt="image"/>

We set the image’s src attribute to {imagePath} to preview it in the browser before processing it.

Converting Selected Images To Texts

As we have grabbed the path to the image selected, we can pass the image’s path to Tesseract.js to extract texts from it.

 import { useState} from 'react'; import Tesseract from 'tesseract.js'; import './App.css';  function App() {   const [imagePath, setImagePath] = useState("");   const [text, setText] = useState("");    const handleChange = (event) => {     setImagePath(URL.createObjectURL(event.target.files[0]));   }    const handleClick = () => {      Tesseract.recognize(       imagePath,'eng',       {          logger: m => console.log(m)        }     )     .catch (err => {       console.error(err);     })     .then(result => {       // Get Confidence score       let confidence = result.confidence        let text = result.text       setText(text);      })   }    return (     <div className="App">       <main className="App-main">         <h3>Actual imagePath uploaded</h3>         <img             src={imagePath} className="App-image" alt="logo"/>            <h3>Extracted text</h3>         <div className="text-box">           <p> {text} </p>         </div>         <input type="file" onChange={handleChange} />         <button onClick={handleClick} style={{height:50}}> convert to text</button>       </main>     </div>   ); }  export default App

We add the function “handleClick” to “App.js and it contains Tesseract.js API that takes the path to the selected image. Tesseract.js takes “imagePath”, “language”, “a setting object”.

The button below is added to the form to call “handClick” which triggers image-to-text conversion whenever the button is clicked.

<button onClick={handleClick} style={{height:50}}> convert to text</button>

When the processing is successful, we access both “confidence” and “text” from the result. Then, we add “text” to the state with “setText(text)”.

By adding to <p> {text} </p>, we display the extracted text.

It is obvious that “text” is extracted from the image but what is confidence?

Confidence shows how accurate the conversion is. The confidence level is between 1 to 100. 1 stands for the worst while 100 stands for the best in terms of accuracy. It can also be used to determine whether an extracted text should be accepted as accurate or not.

Then the question is what factors can affect the confidence score or the accuracy of the entire conversion? It is mostly affected by three major factors — the quality and nature of the document used, the quality of the scan created from the document and the processing abilities of the Tesseract engine.

Now, let’s add the code below to “App.css” to style the application a bit.

.App {   text-align: center; }  .App-image {   width: 60vmin;   pointer-events: none; }  .App-main {   background-color: #282c34;   min-height: 100vh;   display: flex;   flex-direction: column;   align-items: center;   justify-content: center;   font-size: calc(7px + 2vmin);   color: white; }  .text-box {   background: #fff;   color: #333;   border-radius: 5px;   text-align: center; }

Here is the result of my first test:

Outcome In Firefox

The confidence level of the result above is 64. It is worth noting that the gift card image is dark in color and it definitely affects the result we get.

If you take a closer look at the image above, you will see the pin from the card is almost accurate in the extracted text. It is not accurate because the gift card is not really clear.

Oh, wait! What will it look like in Chrome?

Outcome In Chrome

Ah! The outcome is even worse in Chrome. But why is the outcome in Chrome different from Mozilla Firefox? Different browsers handle images and their colour profiles differently. That means, an image can be rendered differently depending on the browser. By supplying pre-rendered image.data to Tesseract, it is likely to produce a different outcome in different browsers because different image.data is supplied to Tesseract depending on the browser in use. Preprocessing an image, as we will see later in this article, will help achieve a consistent result.

We need to be more accurate so that we can be sure we are getting or giving the right information. So we have to take it a bit further.

Let’s try more to see if we can achieve the aim in the end.

Testing For Accuracy

There are a lot of factors that affect an image-to-text conversion with Tesseract.js. Most of these factors revolve around the nature of the image we want to process and the rest depends on how the Tesseract engine handles the conversion.

Internally, Tesseract preprocesses images before the actual OCR conversion but it doesn’t always give accurate results.

As a solution, we can preprocess images to achieve accurate conversions. We can binarise, invert, dilate, deskew or rescale an image to preprocess it for Tesseract.js.

Image pre-processing is a lot of work or an extensive field on its own. Fortunately, P5.js has provided all the image preprocessing techniques we want to use. Instead of reinventing the wheel or using the whole of the library just because we want to use a tiny part of it, I have copied the ones we need. All the image preprocessing techniques are included in preprocess.js.

What Is Binarization?

Binarization is the conversion of the pixels of an image to either black or white. We want to binarize the previous gift card to check whether the accuracy will be better or not.

Previously, we extracted some texts from a gift card but the target PIN was not as accurate as we wanted. So there is a need to find another way to get an accurate result.

Now, we want to binarize the gift card, i.e. we want to convert its pixels to black and white so that we can see whether a better level of accuracy can be achieved or not.

The functions below will be used for binarization and it is included in a separate file called preprocess.js.

function preprocessImage(canvas) {     const ctx = canvas.getContext('2d');     const image = ctx.getImageData(0,0,canvas.width, canvas.height);     thresholdFilter(image.data, 0.5);     return image;  }   Export default preprocessImage

What does the code above do?

We introduce canvas to hold an image data to apply some filters, to pre-process the image, before passing it to Tesseract for conversion.

The first preprocessImage function is located in preprocess.js and prepares the canvas for use by getting its pixels. The function thresholdFilter binarizes the image by converting its pixels to either black or white.

Let’s call preprocessImage to see if the text extracted from the previous gift card can be more accurate.

By the time we update App.js, it should now look like the code this:

import { useState, useRef } from 'react'; import preprocessImage from './preprocess'; import Tesseract from 'tesseract.js'; import './App.css';  function App() {   const [image, setImage] = useState("");   const [text, setText] = useState("");   const canvasRef = useRef(null);   const imageRef = useRef(null);    const handleChange = (event) => {     setImage(URL.createObjectURL(event.target.files[0]))   }    const handleClick = () => {      const canvas = canvasRef.current;     const ctx = canvas.getContext('2d');      ctx.drawImage(imageRef.current, 0, 0);     ctx.putImageData(preprocessImage(canvas),0,0);     const dataUrl = canvas.toDataURL("image/jpeg");      Tesseract.recognize(       dataUrl,'eng',       {          logger: m => console.log(m)        }     )     .catch (err => {       console.error(err);     })     .then(result => {       // Get Confidence score       let confidence = result.confidence       console.log(confidence)       // Get full output       let text = result.text        setText(text);     })   }    return (     <div className="App">       <main className="App-main">         <h3>Actual image uploaded</h3>         <img             src={image} className="App-logo" alt="logo"            ref={imageRef}             />         <h3>Canvas</h3>         <canvas ref={canvasRef} width={700} height={250}></canvas>           <h3>Extracted text</h3>         <div className="pin-box">           <p> {text} </p>         </div>         <input type="file" onChange={handleChange} />         <button onClick={handleClick} style={{height:50}}>Convert to text</button>       </main>     </div>   ); }  export default App

First, we have to import “preprocessImage” from “preprocess.js” with the code below:

import preprocessImage from './preprocess';

Then, we add a canvas tag to the form. We set the ref attribute of both the canvas and the img tags to { canvasRef } and { imageRef } respectively. The refs are used to access the canvas and the image from the App component. We get hold of both the canvas and the image with “useRef” as in:

const canvasRef = useRef(null); const imageRef = useRef(null);

In this part of the code, we merge the image to the canvas as we can only preprocess a canvas in JavaScript. We then convert it to a data URL with “jpeg” as its image format.

const canvas = canvasRef.current; const ctx = canvas.getContext('2d');  ctx.drawImage(imageRef.current, 0, 0); ctx.putImageData(preprocessImage(canvas),0,0); const dataUrl = canvas.toDataURL("image/jpeg");

“dataUrl” is passed to Tesseract as the image to be processed.

Now, let’s check whether the text extracted will be more accurate.

Test #2

The image above shows the result in Firefox. It is obvious that the dark part of the image has been changed to white but preprocessing the image doesn’t lead to a more accurate result. It is even worse.

The first conversion only has two incorrect characters but this one has four incorrect characters. I even tried changing the threshold level but to no avail. We don’t get a better result not because binarization is bad but because binarizing the image doesn’t fix the nature of the image in a way that is suitable for the Tesseract engine.

Let’s check what it also looks like in Chrome:

We get the same outcome.

After getting a worse result by binarizing the image, there is a need to check other image preprocessing techniques to see whether we can solve the problem or not. So, we are going to try dilation, inversion, and blurring next.

Let’s just get the code for each of the techniques from P5.js as used by this article. We will add the image processing techniques to preprocess.js and use them one by one. It is necessary to understand each of the image preprocessing techniques we want to use before using them, so we are going to discuss them first.

What Is Dilation?

Dilation is adding pixels to the boundaries of objects in an image to make it wider, larger, or more open. The “dilate” technique is used to preprocess our images to increase the brightness of the objects on the images. We need a function to dilate images using JavaScript, so the code snippet to dilate an image is added to preprocess.js.

What Is Blur?

Blurring is smoothing the colors of an image by reducing its sharpness. Sometimes, images have small dots/patches. To remove those patches, we can blur the images. The code snippet to blur an image is included in preprocess.js.

What Is Inversion?

Inversion is changing light areas of an image to a dark color and dark areas to a light color. For example, if an image has a black background and white foreground, we can invert it so that its background will be white and its foreground will be black. We have also added the code snippet to invert an image to preprocess.js.

After adding dilate, invertColors and blurARGB to “preprocess.js”, we can now use them to preprocess images. To use them, we need to update the initial “preprocessImage” function in preprocess.js:

preprocessImage(...) now looks like this:

function preprocessImage(canvas) {   const level = 0.4;   const radius = 1;   const ctx = canvas.getContext('2d');   const image = ctx.getImageData(0,0,canvas.width, canvas.height);   blurARGB(image.data, canvas, radius);   dilate(image.data, canvas);   invertColors(image.data);   thresholdFilter(image.data, level);   return image;  }

In preprocessImage above, we apply four preprocessing techniques to an image: blurARGB() to remove the dots on the image, dilate() to increase the brightness of the image, invertColors() to switch the foreground and background color of the image and thresholdFilter() to convert the image to black and white which is more suitable for Tesseract conversion.

The thresholdFilter() takes image.data and level as its parameters. level is used to set how white or black the image should be. We determined the thresholdFilter level and blurRGB radius by trial and error as we are not sure how white, dark or smooth the image should be for Tesseract to produce a great result.

Test #3

Here is the new result after applying four techniques:

The image above represents the result we get in both Chrome and Firefox.

Oops! The outcome is terrible.

Instead of using all four techniques, why don’t we just use two of them at a time?

Yeah! We can simply use invertColors and thresholdFilter techniques to convert the image to black and white, and switch the foreground and the background of the image. But how do we know what and what techniques to combine? We know what to combine based on the nature of the image we want to preprocess.

For example, a digital image has to be converted to black and white, and an image with patches has to be blurred to remove the dots/patches. What really matters is to understand what each of the techniques is used for.

To use invertColors and thresholdFilter, we need to comment out both blurARGB and dilate in preprocessImage:

function preprocessImage(canvas) {     const ctx = canvas.getContext('2d');     const image = ctx.getImageData(0,0,canvas.width, canvas.height);     // blurARGB(image.data, canvas, 1);     // dilate(image.data, canvas);     invertColors(image.data);     thresholdFilter(image.data, 0.5);     return image; }

Test #4

Now, here is the new outcome:

The result is still worse than the one without any preprocessing. After adjusting each of the techniques for this particular image and some other images, I have come to the conclusion that images with different nature require different preprocessing techniques.

In short, using Tesseract.js without image preprocessing produced the best outcome for the gift card above. All other experiments with image preprocessing yielded less accurate outcomes.

Issue

Initially, I wanted to extract the PIN from any Amazon gift card but I couldn’t achieve that because there is no point to match an inconsistent PIN to get a consistent result. Although it is possible to process an image to get an accurate PIN, yet such preprocessing will be inconsistent by the time another image with different nature is used.

The Best Outcome Produced

The image below showcases the best outcome produced by the experiments.

Test #5

The texts on the image and the ones extracted are totally the same. The conversion has 100% accuracy. I tried to reproduce the result but I was only able to reproduce it when using images with similar nature.

Observation And Lessons

  • Some images that are not preprocessed may give different outcomes in different browsers. This claim is evident in the first test. The outcome in Firefox is different from the one in Chrome. However, preprocessing images helps achieve a consistent outcome in other tests.
  • Black color on a white background tends to give manageable results. The image below is an example of an accurate result without any preprocessing. I also was able to get the same level of accuracy by preprocessing the image but it took me a lot of adjustment which was unnecessary.

The conversion is 100% accurate.

  • A text with a big font size tends to be more accurate.

  • Fonts with curved edges tend to confuse Tesseract. The best result I got was achieved when I used Arial (font).
  • OCR is currently not good enough for automating image-to-text conversion, especially when more than 80% level of accuracy is required. However, it can be used to make the manual processing of texts on images less stressful by extracting texts for manual correction.
  • OCR is currently not good enough to pass useful information to screen readers for accessibility. Supplying inaccurate information to a screen reader can easily mislead or distract users.
  • OCR is very promising as neural networks make it possible to learn and improve. Deep learning will make OCR a game-changer in the near future.
  • Making decisions with confidence. A confidence score can be used to make decisions that can greatly impact our applications. The confidence score can be used to determine whether to accept or reject a result. From my experience and experiment, I realized that any confidence score below 90 isn’t really useful. If I only need to extract some pins from a text, I will expect a confidence score between 75 and 100, and anything below 75 will be rejected.

In case I am dealing with texts without the need to extract any part of it, I will definitely accept a confidence score between 90 to 100 but reject any score below that. For example, 90 and above accuracy will be expected if I want to digitize documents such as cheques, a historic draft or whenever an exact copy is necessary. But a score that is between 75 and 90 is acceptable when an exact copy is not important such as getting the PIN from a gift card. In short, a confidence score helps in making decisions that impact our applications.

Conclusion

Given the data processing limitation caused by texts on images and the disadvantages associated with it, Optical Character Recognition (OCR) is a useful technology to embrace. Although OCR has its limitations, it is very promising because of its use of neural networks.

Over time, OCR will overcome most of its limitations with the help of deep learning, but before then, the approaches highlighted in this article can be utilized to deal with text extraction from images, at least, to reduce the hardship and losses associated with manual processing — especially from a business point of view.

It is now your turn to try OCR to extract texts from images. Good luck!

Further Reading

Articles on Smashing Magazine — For Web Designers And Developers


Deprecated: Function create_function() is deprecated in /home2/thebrand/public_html/wp-content/plugins/wp-spamshield/wp-spamshield.php on line 1884
paul-boag-300x300-1
24 Mar

Smashing Podcast Episode 12 With Paul Boag: What Is Conversion Optimisation?

by TBSCategories: News

Smashing Podcast Episode 12 With Paul Boag: What Is Conversion Optimisation?

Smashing Podcast Episode 12 With Paul Boag: What Is Conversion Optimisation?

Drew McLellan

Paul BoagIn this episode of the Smashing Podcast, we’re talking about the user experience around converting site visitors into customers. Can our selling techniques leave customers feeling cheated? And how can that be avoided? I spoke to conversion optimisation specialist Paul Boag to find out.

Show Notes

Weekly Update

Transcript

Drew McLellan: He’s a user experience consultant, digital transformation expert, and conversion rate optimization specialist hailing from the southwest of the UK. He is the author of many books, including User Experience Revolution from Smashing Magazine, and the forthcoming title Click! How to Encourage Clicks Without Shady Tricks. He’s also a veteran web design podcaster with a show running more or less weekly for 15 years. So we know he’s an expert at user experience design, but did you know he’s also the world authority on papier-mache? My Smashing friends, please welcome Paul Boag. Hi, Paul. How are you?

Paul Boag: Oh, I’m smashing, apparently.

Drew: So we’re in the middle of a global pandemic.

Paul: Yeah, and you’ve just made me say I’m feeling smashing in the middle of a pandemic. That’s great. Thanks.

Drew: What I want to know is, what’s on your shopping list?

Paul: Oh, dear. Do you know, this pandemic has changed very little in my life. Social distancing, I’ve been doing that for 18 years. I’ve worked from home for 18 years. We homeschooled our son. My wife works from home. Nothing has changed in my life. In fact, if anything, I’m now more sociable because everybody’s creating these WhatsApp groups and things, where, “Oh, let’s all pull together as a community.” So I’m having to talk to people now. It’s just dreadful, dreadful time.

Drew: Oh, wow. I’ll tell you what’s on my shopping list.

Paul: Oh, yeah.

Drew: It’s a new book of yours.

Paul: Oh. That was smooth, Drew. I’m super impressed by that.

Drew: It’s a heck of a title, How to Encourage Clicks Without Shady Tricks. Tell me about that.

Paul: Well, that wasn’t the original title that I wanted. Do you know this story?

Drew: No, I don’t know this story.

Paul: Oh, okay. I got vetoed by Vitaly.

Drew: Oh, dear.

Paul: Because I wanted to call it Encouraging Clicks Without Being a… But apparently, that’s not professional enough. So the basis of the book is that we all need to improve our conversion rate. Websites aren’t there, although we talk about being user-centric and user-focused, and all those things are entirely correct, but at the end of the day, websites have to create and return on investment for whoever owns them. And that’s entirely understandable and as it should be. But increasingly, people are under enormous pressure to improve their conversion rates. Marketers have got targets they’ve got to meet, designers are under pressure. And part of the problem is, in the early days when your website is rubbish, it was easy to increase your conversion rate.

Paul: And so as a result, then that set expectations, because the conversion rate went up quite a lot every year. And so management ended up expecting that to happen over the long term, and of course, it becomes harder and harder. So the result is that people are inevitably turning to dark patterns. Not because necessarily they want to, but because they’re under pressure to. They’re under pressure to bring about results.

Paul: So the premise of this book is, first of all, to explain why dark patterns are a bad idea. And not from an ethical point of view. I don’t feel I’m the kind of person that can preach on ethics. But I talk about it from a purely business point of view, that these are the business reasons why dark patterns are ultimately damaging. And then that inevitably leads to the question of, well, if dark patterns aren’t the answer, then what is? And that’s what the majority of the book is exploring, that, what you can do to improve your conversion rate without resorting to these kinds of more manipulative techniques. So I actually got really excited about this book. It’s been one of the most enjoyable ones to write, and I actually think it’s probably the most practical and tangible book for the biggest majority of people out of the ones I’ve written.

Drew: So in a previous episode of this podcast, I talked to Michael and Trina about their book on ethical design.

Paul: Yes.

Drew: Also from Smashing magazine.

Paul: Yeah.

Drew: I guess that makes your book a good complement to that one, if they’re looking at the ethics of it, and maybe your approach is slightly more from the practical and business side.

Paul: Absolutely, yeah.

Drew: And not just the ethics.

Paul: Yeah. And I was quite chuffed when I found out that my book would be following on from theirs. I felt that that was a really good relationship between the two. Because don’t get me wrong, I think the ethics of how we design online and the decisions that we make and those kinds of things is massively important. And I think we’re in a very dangerous time in terms of that kind of thing, with many of the decisions that are being made, especially by larger organizations. But that’s not an area that I am an expert in, or that I feel I can comment on. But what I am seeing are significant long-term consequences of adopting these more manipulative techniques. Because I think there’s a perception that users are unaware that they’re being manipulated. Because these techniques work, people go, “Oh, okay, therefore people aren’t aware that we’re manipulating them. So everything’s fine.”

Paul: The truth is very different from that. Those things, these forms of manipulation, are happening on a subconscious level, yes. And they work because they’re subconscious, but people are still consciously aware of it. I’ve done a lot of usability testing on things like hotel booking sites, which are well known for this kind of thing. And the truth is, people will go, “Oh, I hate all this manipulative crap.” But then they’re still influenced by it. So it’s the impact of the fact that people are aware of it, and then also it’s got a lot of hidden costs associated with it. You tend to see high returns, you tend to see more contacting of support and those kinds of things. And a lot of organizations are not joined up enough to realize that that kind of thing is happening because of these dark patterns.

Drew: I guess it’s a bit like going to a theme park and buying lunch there. You know that you’re being way overcharged, but you want your lunch at the theme park, and so it leaves, not literally, a bad taste in your mouth, hopefully. But there’s a bit of that regret there, that you know that you’re being shafted but you go along with it all the same. But you might not be keen to return next time you’re budgeting your holiday.

Paul: Absolutely. But then there’s also the element of buyer’s remorse, that a lot of time if you bounce someone … You can bounce people into doing pretty much anything you want. And that’s fine, take out ethics for a minute. But you could argue, but that’s fine from a business point of view. But what you end up with is an audience of customers who are suffering from buyer’s remorse. Buyer’s remorse is extremely dangerous because that’s what leads people to complain online, that’s what leads people to return items and all of those kinds of things. So it’s incredibly important that people are happy with their decision, they’re happy with their purchase. And that’s really what the book is about, is how do you get people to a point as fast as possible where they’re happy with their decision without them then afterwards regretting that decision?

Drew: So say that you’ve been using some of these perhaps slightly underhanded techniques on your site, and you’ve seen that it’s converting well, you’re turning visitors into customers quite effectively. But you want to, perhaps you’re seeing more returns or you’re seeing some bad reviews, or you’re seeing some of the consequences of this buyer’s remorse that we were talking about, and you want to try and improve things, get rid of the underhanded techniques, and bring in a more ethical or more customer-friendly ways of converting people. How are you going to know if you’re actually having an effect? You presumably have to come up with some way of measuring your conversion before you can start making changes?

Paul: Yeah, absolutely. And I think, again, this is something that many organizations are quite poor at, is being clear about how they’re going to measure success. Now, yes, your conversion rate is one metric you should absolutely be following. But even with conversion, you need to be a little bit more sophisticated than how many people are buying. You also need to pay attention to average order value, you need to pay particular attention to lifetime value. So how much are customers worth over their entire life? Because you may well find that you’re getting quite high churn of customers if you’re using dark patterns and things like that.

Paul: But really, conversion should be just one of the metrics that you’re measuring. There’s also things like, you need to be paying attention to engagement, how engaged are people with your content. Because that makes a big difference in whether they eventually go on to convert. So you’re looking at things like, how much of your videos do they watch? How long do they spend on your site and what are they looking at while they’re doing it? Are they engaging on social media and those kinds of things? And then the final aspect is obviously usability. You need to be measuring how quickly someone can complete a particular task on their website, and how easy they find the system to use, and various different other criteria.

Paul: And there are loads of mechanisms that you can use for measuring those different things. There are some great tools out there. And also some good metrics that you can adopt. So, for example, with usability, there’s something called the system usability scale, which could be a very useful metric to measure. But equally, there are tools like maze.design, is one that I often use, which will measure how long it’s taking someone to make a purchase, for example, get through the checkout. So yeah, having that broad range of metrics, you’re not just focusing on how many sales did we make this quarter. You’ve got to look at the bigger picture.

Drew: Are there any dangers that you could end up measuring the wrong thing?

Paul: Yes. Yes, absolutely. And so as a result, I think any metric that you’re looking at, that’s why you really need a range of metrics. If you just focus in on one particular metric, so for example, a lot of marketing professionals are judged on the number of leads that they’ve generated in a particular quarter or whatever. If you just have one metric like that, then it’s going to, A, skew the reality of what’s going on, but B, it’s going to end up encouraging some less than perfect behavior. So let me give you an example. I worked with a company that produces project management software. And they had a marketing department and they had a sales department. And the marketing department was tasked with generating leads. That was their job. They had to generate as many leads as possible.

Paul: The sales department were judged and assessed on how many leads they converted. Now, the website was owned by the marketing department. So the marketing department came to the conclusion that it was a really good idea to, the product demo that they had on the website, they were going to make people register before they could see the product demo. Because that would generate a large number of emails and a large number of leads to help them meet their target. Sure enough, it did. It generated a huge number of leads, a lot of people just gave up and went away, but many people did actually sign up to see the demo.

Paul: Now, the problem that that created is, the majority of those leads weren’t good quality leads. They were people that were nowhere near the point of being ready to talk to a salesperson, and so when the salesperson contacted them, they weren’t interested. They didn’t want to talk to them. But the salesperson had already wasted the time and the effort in calling them. And then they also had to filter through all the people that entered “Donald Duck@Disney” as their email address. So actually they created a huge amount of internal work for the sales team, and the sales team conversion rate plummeted through the floor. Because they had all these poor quality leads. So that’s a great example of how things can go wrong if your metrics are too narrow and too skewed in a particular direction.

Drew: And I guess a lot of it comes down to understanding who the user is. In order to turn somebody into a customer, you need to understand who they are. I guess so much about user experience design is effectively putting yourself in the place of your user and empathizing with what they’re trying to do. So how do we go about understanding who our user is?

Paul: Well, again, there are lots of different ways of doing that, depending on your time and your budget and things like that. I’m a great believer in actually meeting with users. I think there is a bit of a tendency at the moment towards, we’ve got all these wonderful analytics and survey tools and that kind of thing. And sure enough, those are great. I’m not in any way knocking them. But if you’re trying to convince somebody to take action, as you say, you need to empathize with them. And knowing that somebody has 2.3 children, works in the city, and I don’t know, spends their weekends kayaking, doesn’t really help you to know and empathize with them as people.

Paul: So personally, I’m a much bigger fan of actually speaking to people and doing user interviews. Even if they’re over the phone or remotely, which they have to be at the moment. They’re very much worthwhile in terms of getting to know people. Now, that said, I think another thing that I love to do, and just kind of blows you away, you can’t do this at the moment, but hopefully you will be able to soon, which is, you go and visit people in their homes. Now, the reason that I find this so valuable is because you find out about the reality of their experience in a way that you would never get from just talking to them over the phone or from a survey.

Paul: Now, I’m not saying you need to do this a lot. Probably doing it once is enough. But essentially … Let me give you an example. I was once testing an e-commerce site, and so I decided to visit some people in their home and actually test the site with them in their own home. So I went to visit this one lady, and she opened the door, and immediately all these cats came out and crawled around my legs and disappeared off. And she was a stereotypical cat lady. I’m sorry to be rude, but she really was. Every surface in the house was covered with knickknacks and ornaments relating to cats in some way. There were pictures of cats on the wall. She had a total of nine cats, which is just insane.

Paul: So we talked for a while, and we sat down to use this, to test with this website. And we used her computer, which I tell you took 10 minutes to boot up because it was this old tower computer, it was an absolute nightmare. And that whole desk was covered with clutter and knickknacks and things. And she had post-it notes all around her screen. Now, the minute she sat down in front of that computer, a cat jumped on her lap. So she spent the rest of that usability session trying to juggle a cat that was on her lap. The cat got so fed up at one point that it wasn’t getting enough attention that it decided to crawl across the keyboard and sit on the keyboard. Which I know you understand because you’ve got a cat and I’ve seen pictures of your cat doing the same thing.

Drew: I feel seen.

Paul: Yeah, exactly. And then, on top of which, I asked her at one point, “Add a product to the basket,” which she was able to do. And then I said, “Now, go to the basket,” and I realized there’s no way she’s going to find the basket. Because she had a post-it note stuck on the screen over the top of the basket. Now, that is the real-world experience. We sit down and look at our websites and we think they’re so easy to use and so simple. But if you’re juggling with a cat and if you’re living in chaos, or you’ve got distractions and that kind of stuff, then you don’t have that clean, clear mental point of view to approach the website. You are under what is called cognitive load.

Drew: So I guess one solution to that might be for every design studio to be equipped with cats.

Paul: Yeah, absolutely. And not only that, but equipped with (bad) computers. Because we all sit, don’t we? I’m doing it right now, I’m sitting in front of a lovely iMac with this gorgeous screen. And that’s not what most people … I remember another occasion I was designing a website for an elderly audience. Well, I say I was designing it, I was doing the UX side of things. And I was getting quite frustrated with the designer, because the designer was a young guy, and he did these beautiful interfaces that was all subtle and lovely and gorgeous. And I was saying, “I’m sorry, but this old audience are not going to be able to see what you’re doing, and they’re not going to be able to click on all these little links and stuff. And you’ve got to make it more brash and horrible,” which obviously he didn’t want to do.

Paul: So I came into the office one day with a pair of reading glasses and a pair of ski gloves. And I made him put on the reading glasses and put on the ski gloves. Now, he didn’t need reading glasses, just to be clear on that. And I said, “Now use the site.” And, of course, he couldn’t. He couldn’t see properly, he had lost motor control in his hands because he had these big, thick ski gloves on. So that helped him to empathize and understand what his audience was going through. And so things like that, I think, are really important to do. Obviously, that’s a bit of an extreme example, it was me making a point and rubbing my designer’s nose in it. But you get the idea.

Drew: You mentioned the cognitive load of buying. Making personal decisions online can be fairly overwhelming at times. Are there things that we should be doing that are going to help the customer have a better experience and be more likely to convert, more likely to actually make a purchasing decision?

Paul: Yeah. It’s really funny, cognitive load is a fascinating thing in terms of how it affects us. So what cognitive load is basically, is having to think too much. We have two systems in our brains. We have system one and system two. And system one is that unconscious decision making that we make all the time, and system two is our conscious mind. Now, our conscious mind is incredibly powerful, but it easily gets overwhelmed. It easily gets tired out. And that’s known as cognitive load. And when we are overwhelmed, when our cognitive load goes up, it has all kinds of really bad impacts on conversion rate. So suddenly your website feels hard to use, it feels untrustworthy and a little bit suspicious, and it feels just downright bad. So our conscious minds are very cynical as well, so we start not believing what’s being said on the site. And it really has an enormous impact.

Paul: So the way that you lower cognitive load is really about simplifying your website. So removing unnecessary distractions that are on the website. It’s about being consistent in your website. So buttons don’t move around, things don’t change. But not only consistent within your own website, consistent with people’s expectations from a website. So to give you an example of what I mean, let’s say I asked you to search on a website. Where would you look? You’d look in the top right-hand corner, wouldn’t you?

Drew: Right.

Paul: Everybody looks at the top right-hand corner, so the first thing, they look at the top right-hand corner, then they look for an input field. They enter their search query in the input field, and they press the button next to it. That’s what’s called procedural knowledge. We have learned that that is a procedure that, if we go to a website, look at the top right-hand corner, use the input field, click the button, we will search on that website. But the minute that you break that procedural knowledge, our cognitive load goes up. So at that point, we’re starting to go, “Well, hang on, where’s the button?” Or, “Why isn’t the search field the way it’s supposed to look?” Or, “Why isn’t it in the top right-hand corner?” So that consistency with expectations matters a lot.

Paul: And then our mood matters as well. Bizarrely, you know what it’s like, some days you wake up and you’re in a really good mood. And everything seems to be easier. And then other days you wake up in a bad mood, everything seems to be harder. So actually, the mood we’re in affects our level of cognitive load. So things like design delighters, nice little friendly copy, colorful graphics, all those things, they help as well to put us in a good mood that lowers our cognitive load. So it’s really mood, inconsistencies in the interface, confusing, too much information being displayed, and then finally priming people. So in other words, setting their expectations about what will happen and why it will happen.

Drew: I guess all these things weigh into how much trust somebody has in the website. And I think trust is quite a big factor, isn’t it, in inciting someone to actually buy from you?

Paul: Oh, enormous.

Drew: Because anyone can build a website online, we all know that. We’re web people. And there are thousands of places that you can buy the same product or service from, quite often. So there needs to be some way of building up trust. Are there other ways that we can build up trust in a site?

Paul: Yeah. You’re right, people buy from people they trust and from organizations they trust. So when it comes to trust, a lot of it is about humanity. Being a human being. You know how many times you go to a website and it just feels like it’s spouting marketing BS at you? Do you know what I mean? The great example I use in a book, where there’s some copy on a university website. I think it was the University of Essex website. And basically, that copy was written in the third person. So it’s talking about the reader as students, it was talking about themselves as the University of Essex. And it just felt, it lacked any kind of humanity and any sense that they cared about you or liked you. And simply turning that around and writing it in the first person, so talking about “we” and talking about “you,” makes an enormous difference in making that connection with people.

Paul: Other ways that you can do that, obviously, is through things like social proof. You can build trust by demonstrating that you’re a trustworthy brand. But again, you’ve got to accept that people are very cynical these days. So it’s not a matter of just slapping some logos on your website and going, “There’s your social proof.” Or slapping a text testimonial. Because people know that some companies just make that stuff up. So one of the things, actually a product of your own, actually, Perch, I don’t think you do this anymore, but for a long time on your website, you used to have testimonials that came straight from Twitter. They were pulled straight out of Twitter. And that was such a good idea, because those testimonials, people can click through and see that there’s a real human being behind it.

Paul: Another example is videos. If you’ve ever watched video testimonials on websites, where the person that’s speaking is beautifully lit, has got perfect teeth, and says a whole spiel about how great the company is without pausing, without going, “um,” without making mistakes, they’re obviously just reading off a script. That’s insincere. However, if you have a shitty video that’s filmed on a webcam, that the audio’s a bit crap, that’s actually more effective. Because it’s more real. A great example of this is: there’s a guy called Paul Jarvis who sells an online course called Chimp Essentials, which teaches you how to use MailChimp. And if you go to chimpessentials.com, he’s got all these videos, just like he’s obviously just chatting to someone over a webcam and them talking about how great the course was and how much it helped. And that is so powerful, because it’s authentic, it’s real, it’s human. And that’s what it’s about. It’s about coming across as a human being and making a connection with people.

Paul: So yeah, I’m a huge fan of that kind of sincerity and openness and honesty. Buffer is another example of that. Buffer, they lay out everything online. You can see how much their employees are paid, you can see how much they earned in the last month, you can see what their diversity range of staff and employees are. You can see everything about that company. And so you know they’re not hiding anything. You know that they’re being sincere and open. And that goes such a long way.

Drew: So we understand who our customer is. We want them to buy from us. Is there a single moment when a conversion happens on our site?

Paul: No. No, conversion is very much a journey. So it’s a series of micro-interactions. And I think that’s part of the problem that people have, is that they perceive, “Okay, they’ve clicked on the ‘buy now’ button, we’re done.” Or they’ve sent us a contact form, we’re done. And actually it’s a lot more nuanced than that. There are many, many steps in a journey that someone goes on. So let’s take, I don’t know, donating to charity, for example. So the first interaction might be, you see an update that a friend shares on social media about some crisis in the world. So your first point of conversion really is just clicking on that link in that social media update, which takes you to a blogpost. And then your second conversion is actually reading that blogpost and looking at it.

Paul: And your third, maybe, is to sign up for updates about that particular crisis. And then maybe you receive some emails about it. So your next conversion point is actually opening those emails and reading the content. And then it might be that they, in those emails, ask you to make a donation. So the next step is to make a single donation, one-time donation. And then they’ll follow up with you again, and that might turn into a monthly donation. And then that might lead you to actually start volunteering for them or fundraising for them. And, eventually, it might even lead to you leaving a legacy when you die.

Paul: So actually, any point of conversion is just a step in an ongoing journey. And we need to start thinking of conversion as a journey of relationship with a customer, rather than, “Okay, we’ve got them to sign on the dotted line, we’re done.” Because there’s so much more potential than just that.

Drew: So every interaction you have with that customer is a point where really you’re selling with them. It’s always be closing, right? ABC.

Paul: Yeah. Yes, you know, I hadn’t thought of that. Yes, it’s absolutely straight off of that, isn’t it? Yeah.

Drew: So you’ve carefully designed a site, you’ve done your best to ensure that you’re going to get some good conversions without any dirty tricks, without anything underhand. You launch the site. Is that it? Are you done?

Paul: No.

Drew: What should we be doing over time?

Paul: You’re asking questions that I know you know the answers to. That’s the sign of a great interviewer, when you’re happy to ask questions that other people might need the answer. No, of course it is. You’ve run Perch for years, you know that you can’t just start something and then walk away from it. When it comes to conversion, really that initial launching is the very, very beginning of the process. And actually, what is important in the journey is that ongoing optimization, that ongoing monitoring and improvement. It’s interesting, I’ve literally just written a report for a company outlining this approach. So basically it begins, you go through a cycle. And the cycle begins with, you try and identify problem areas in your site. So typically you would look at your analytics. And you’d look at pages that have got a high dropout rate for the number of visitors that they have to that particular page. So a high percentage of dropouts. So you know that something is going wrong on that page. So that’s your first step. Then you try and narrow down, well, what’s going wrong on that page?

Paul: Now, you can either do that through session recorders, which allows you to watch back users interacting with that particular page, and you might be able to spot where it’s going wrong. Perhaps you’re having problems with a form or perhaps they scroll straight past some critical button, or something else. Alternatively, if that doesn’t work, you might want to do some remote usability testing, and that’s so cheap and easy these days, using services like usertesting.com or User Brain. There are all these great services that make it really easy to test those kinds of pages. So that will enable you, probably, to work out what the problem is. Once you’ve done that, then the next step is, now we need to identify how we’re going to fix that problem. And in that particular case, you can have a couple of different approaches.

Paul: If the problem is relatively easy to fix, so if it’s just things like changing imagery or changing button colors or that kind of very simple thing, then probably you’ll want to go straight to A/B testing. So you use something like Google Optimize to send some of your users to an alternative version of the page with these small changes being made. You see whether that improves things. And if it does, you push it live. If it’s more substantial changes, then you’ll probably want to mock that up or prototype that before you go to all the expense and time of building it properly. And then you can test that either by doing some more usability testing, or if it’s just a static mockup, in other words, it’s not an interactive thing, then you might want to do something like a first-click test, where you say, “Okay, where would you click on this mockup in order to complete X task?” And see whether people are clicking in the right place.

Paul: Or another test that you can do is something called a five-second test, where you show them the mockup for five seconds, take it away, and you ask them what they recalled. And that’s a really good way for ascertaining whether or not they spotted that call to action they previously scrolled straight past. So you have to use a bit of imagination, but basically you then test your hypothesis, if you like, about how to fix it. Presuming you’re right, you then roll it out. Now, you’re not at the end of the process then. Because you now go back to the beginning again. You look at, “Well, what’s the next page that people are most exiting the site from?” And you repeat the process. And you just keep doing that, really, forever, slowly and incrementally improving your conversion rate over time.

Drew: This all sounds very logical, very attractive to somebody who, like me, who’s a business owner and I’ve got digital product, and I think, yes, I could be doing this. Some of the listeners will be freelancers working on an hourly rate for a customer who has these sorts of problems. And I can see it might be difficult to convince that customer that this is what they should be doing, this is something they should invest in. Because all the time spent in even just setting up some monitoring and setting up something to measure conversion is money that that client is going to have to pay out for the designer to do it. What would you say to those designers? Is there an effective way they can persuade their client that this is what they should be doing?

Paul: Yeah. This is a tricky one because it’s very hard to persuade anybody of anything, in my opinion. People become very entrenched in their ways of thinking. But it’s worth finding out what they care about. Because there will be certain metrics that they want to move. It might be they want more leads, it might be they want more quality leads, it might be that they want to reduce their cost of sale. Whatever it is. And then you start talking about these kinds of optimization techniques in that context. That said, you’re never going to get someone to commit to a major program of incremental design and usability testing and all those things in one go. What I would start with is just trying to encourage them to go through that cycle once. So it might be that you even have to install the analytics on your dime. But it’s not a big deal, is it, to add something like Google Analytics or something like Hotjar for session recorders.

Paul: So you basically then just take them through it once. You say, “Okay, well, let’s give it a go for one time. Let’s see if we can make a difference to the conversion rate by looking at the analytics, by mocking up one thing, and just improving that one thing.” If that works, you rinse and repeat. They don’t necessarily need to commit to an ongoing retainer where you’re continually optimizing it. This could be done as a series of sprints, effectively, going through the loop one thing at a time. In terms of more general stuff, like for example, just simplifying the interface, then in those kinds of cases, often it’s a matter of actually showing them what better might look like. Now, I’m afraid to even say this in front of you, Drew, because I remember many, many years ago … Or no, it wasn’t you, actually, I was about to say that it was you I remember going on about speculative design. But it wasn’t. Yeah, you’re off the hook.

Paul: It was you ranting about the fold. That was the thing that I’m remembering. Anyway, that’s beside the point. So one of the things you can do there is actually try … For the first time, I’m not saying do this regularly. But if you’ve got an existing client and you’re seeing glaring problems, mock up an alternative version of doing a particular page, and then do those five-second tests that I mentioned, or do a, what was the other one, a first-click test. And see whether those can make improvements, and show that evidence to the client. So you can use a tool like usabilityhub.com that will allow you to do those tests really easily. If you don’t want to do that because you don’t want to put the effort of mocking those things up, the other thing that you can do is just record a few videos of people using the website. And then edit down all of the bits of them moaning and complaining about how crap the site is to use into a two-minute horror reel, and send it to the client. That can do it as well.

Drew: So would these generally be, would these be the tips you’d give somebody who was just wanting to get some quick wins to start? They’ve done no optimization, and they just want to get started. Are these the same sort of things they’d do for a quick win?

Paul: Yeah, to some extent. Often I find, where I start, if I get a new client, the thing I normally start with is simplifying the user interface. Because inevitably sites end up bloated with a load of things that don’t really help. So the way that I tend to do it is, I’ll take a particular page, let’s say a home page, for example, and I will systematically look at every single element on that page. And I will ask myself three questions in order. Question number one I ask is, could I remove this element? If I removed this element, what would be the negative consequences of doing so? So you’d be amazed at how much you can just strip out of a website. And of course, every element that’s on the screen is every element a user is having to process and look at. Every element adds to their cognitive load. So anything that you can remove, great.

Paul: Then if you can’t remove an element because it’s critical to the completion of a key task or convincing users to take action or whatever else, then the next question you ask yourself is, can I hide this element? Could I move it deeper in the information architecture? Could I put it under a tab? Could I put it under an accordion? Whatever else, just to get it out of the site for the majority of people. So that often helps. And then if the answer is, well, this really is absolutely critical, so I’ll give you an example, maybe you’re forced to have some compliance copy on your website. That happens, I get that a lot with my bigger clients. “Oh, yes, legal department says we have to have this.”

Paul: So my third option, my final option, is can I visually shrink this? So can I de-emphasize it, put it in the footer or make it smaller or make it lighter text? Anything to draw attention away from that and focus it on stuff that really matters. So that’s almost always my first starting point with any kind of conversion, is just to simplify everything. The other thing that I do alongside that, which is a really useful starting point, is called objection handling. So I will make a list of as many reasons that someone might choose not to act as possible. And I then make sure that those are all addressed clearly on the website. Because often they’re not. So a classic example of that might be, somebody who comes to sign up for a newsletter. Now, you think, when you sign up to a newsletter all these things go through your head, are they going to sell me an email address? Are they going to spam me? Is the content going to be relevant? Can I unsubscribe in one click?

Paul: All of these objections. Are you addressing those? And not only are you addressing them, but are you addressing them at the point that you want people to convert? So for example, yes, you probably say something about whether you’re going to sell their email address or how much you’re going to email them and stuff. That’ll be in your privacy policy. But that’s going to be buried in a completely different part of the site where no one ever looks. And to be honest, it’s all illegible anyway because it’s written in legalese. So answering people’s questions at the point they have them is also another really good starting point for just giving your conversion rate that little boost.

Drew: That’s fantastic. And I think all these tips are things that you go into in quite a lot of depth in the book.

Paul: Mm-hmm (affirmative).

Drew: I found it to be a really great read, I’ve enjoyed browsing through it so far. I know it’s in the late stages of production and it’s coming out from Smashing magazine this spring, 2020.

Paul: Mm-hmm (affirmative).

Drew: That’s Click: How to Encourage Clicks Without Shady Tricks. And you can find out more about the book if you go to encouragingclicks.com. And then when it’s released you can go to smashingmagazine.com/books and you’ll be able to find it and buy it from there. Now, Paul, I guess if people are listening to this, I presume that they like podcasts. You’ve been podcasting for I guess almost 15 years now?

Paul: Yeah. Yeah, we were the first web design podcast in the world, that’s my claim to fame.

Drew: There we go. If a listener hasn’t come across the Boagworld podcast, what sort of thing could they expect from it?

Paul: Nonsense, generally. No, it’s a season-based show that covers all aspects of web design in a pretty broad sense. We don’t do a lot on development, to be honest. So it’s mainly design, project management, content creation, all of those kinds of things. A lot of UX stuff, because that’s my personal passion. It’s quite a laid-back conversational show. We have seasons, and different seasons are on different topics. So for example, there’s a whole season dedicated to conversion rate optimization, if people want to learn more about that as a subject area. So you can pick and choose which seasons you listen to. Yeah, I don’t think there’s a huge amount more to say about it. The current season has been one about almost like a virtual meet-up, where we’re having conversations with people listening to the shows. You can come along and join in. But next season will be something totally different. So it’s a bit of an eclectic mix, to be honest.

Drew: I’ve listened to it on and off for a good number of those 15 years, and it’s always an enjoyable listen. So I would recommend that people check it out if they haven’t done so already. So I’ve been learning from you about honest ways to convert users into customers. What have you been learning about lately, Paul?

Paul: I’ve been getting slowly more and more obsessed with psychology. Because obviously that relates quite a lot to both user experience design and conversion rate optimization, so I’ve been realizing how little I know about psychology. So it started off very simply, by following a guy called Joe Leech, who’s written a free e-book that you can download called Psychology for Designers. So that got me thinking more about psychology and understanding how to approach the subject. And then I’ve started to read proper psychology books now. I feel like I’m a bit more of a grownup in the field. So for example, earlier I was talking about system one and system two, that comes from a book called Thinking Fast, Thinking Slow, that I would highly recommend.

Paul: It’s a bit of a stodgy read at places. It’s not the easiest book to read. But it’s very, very much applicable to the kind of world that we’re looking at. So a lot about psychology. I’m also always nosing into things like marketing and sales as well, because I’m quite interested of how people apply sales techniques offline. So all that stuff I was talking about, like when I talked about objection handling, that comes from traditional sales, basically. So yeah, those are the two big areas at the minute.

Drew: If you, dear listener, would like to hear more from Paul, you can follow him on Twitter, where he’s @Boagworld, or find his podcast, blog, and details of how you can hire him to consult on your digital projects at boagworld.com. Thanks for joining us today, Paul. Do you have any parting words?

Paul: Keep safe, I guess, at the moment. That’s the sad thing that you’re having to say, isn’t it? And yeah, just start experimenting with conversion rate optimization. You’ll be amazed at the results you see.

Smashing Editorial (dm, ra, il)


Articles on Smashing Magazine — For Web Designers And Developers

litmus-reading-environment
21 Jun

Design Your Mobile Emails To Increase On-Site Conversion

by TBSCategories: News

Design Your Mobile Emails To Increase On-Site Conversion

Design Your Mobile Emails To Increase On-Site Conversion

Suzanne Scacca

I find it interesting that Google has pushed so hard for web designers to shift from designing primarily for desktop to now designing primarily for mobile. Google’s right… but why only focus on designing websites to appeal to mobile users? After all, Gmail is a leader within the email client ranks, too.

Email can be an incredibly powerful driver of conversions for websites, according to a 2019 report from Barilliance.

On average, emails convert at a rate of 1.48%. That includes all sent emails though — which includes the ones that go unopened as well as the ones that bounce. If you look at emails that are opened by recipients, however, the average conversion rate jumps to 17.75%.

Let’s go even further with this:

Recent data from Litmus reveals that more emails are opened on mobile than on any other device:

Litmus email opens data
Litmus data reveals that 43% of email opens occur on mobile devices. (Source: Litmus) (Large preview)

Many sources even put the average mobile open rate at well over 50%. But whether it’s 43% or 50%+, it’s clear that mobile is most commonly the first device people reach for to check their emails.

Which brings us to the following conclusion:

If users are more likely to open email on mobile and we know that opened emails convert at a higher rate than those that go unopened, wouldn’t it make sense for designers to prioritize the mobile experience when designing emails?

Mobile Email Design Tips to Increase Conversions

Let’s explore what the latest research says about designing emails for mobile users and how that can be used to increase opens, clicks and, later, your website’s conversion rates (on mobile and desktop).

Design the Same Email for Mobile and Desktop

Although email is often ranked as the most effective marketing channel for acquiring and retaining customers, that’s not really an accurate picture of what’s going on.

According to Campaign Monitor, here’s what’s actually happening with mobile email subscribers:

Campaign Monitor mobile CTR data
Campaign Monitor charts the progression from mobile email opens to click-through rate. (Source: Campaign Monitor) (Large preview)

The open rates on mobile are somewhat on par with the Litmus data earlier.

However, it can take multiple opens before the email recipient actually clicks on a link or offer within an email. And guess what? About a third of them make their way over to desktop — where they convert at a higher rate than those that stay on mobile.

As the report states:

Data from nearly 6 million email marketing campaigns suggests the shift to mobile has made it more difficult to get readers to engage with your content, unless you can drive subsequent opens in a different environment.

I’ve reconstructed the graphic above and filled it with the number of people who would take action from an email list of 1,000 recipients:

Campaign Monitor mobile open and click data
An example of how Campaign Monitor’s data translates into real-world numbers. (Source: Campaign Monitor) (Large preview)

At first glance, it looks as though mobile is the clear winner — at least in terms of driving traffic to the website. After the initial mobile open, 32 subscribers go straight to the website. After a few more opens on mobile, 5 more head over there.

Without a breakdown of what the user journey looks like when opened on desktop, though, the calculation of additional clicks you’d get from that portion of the list isn’t so cut-and-dried.

However, let’s assume that Litmus’s estimate of 18% desktop opens is accurate and that Campaign Monitor’s 12.9% click-through rate holds true whether they open the email first on mobile or desktop. I think it’s safe to say that 23 desktop-only email opens can be added to the total.

So, that brings it to:

37 clicks on mobile vs 26 on desktop.

Bottom line: while mobile certainly gets more email subscribers over to a website, the conversion-friendliness of desktop cannot be ignored.

Which is why you don’t want to segment your lists based on device. If you want to maximize the number of conversions you get from a campaign, enable subscribers to seamlessly move from one device to the other as they decide what action to take with your emails.

In other words, design the same exact email for desktop and mobile. But assume that the majority of subscribers will open the email on their mobile device (unless historical data of your campaigns says otherwise). Then, use mobile-first design and marketing tips to create an email that’s suitable for all subscribers, regardless of device.

Factor in Dark Mode When Choosing Your Colors

You don’t want there to be anything that stands in your users’ way when it comes to moving from email to website. That’s why you have to consider how their choice of color and brightness for their mobile screen affects the readability or general appearance of your email design.

There are a number of ways in which this can become an issue.

As we hear more and more about how harmful blue light from our devices can be, it’s no surprise that Dark Mode options are beginning to roll out. While it’s prevalent on desktops right now, it’s mostly in beta for mobile. The same goes for email apps.

That said, smartphone users can hack their own “Dark Mode” of sorts. This type of color inversion can be enabled through the iPhone’s “Accessibility” settings.

Gadget Hacks iPhone 'Dark Mode' hack
Gadget Hacks shows how iPhone users can hack their own ‘Dark Mode’. (Source: Gadget Hacks) (Large preview)

Essentially, what this does is invert all of the colors on the screen from light to dark and vice versa.

Unfortunately, the screenshotting tool on my iPhone won’t allow me to capture the colors exactly as they appear. However, what I can show you is how the inversion tool alters the color of your email design.

This is an email I received from Amtrak last week. It’s pretty standard with its branded color palette and brightly colored notices and CTA buttons:

Amtrak email on Gmail mobile app
What a promotional email from Amtrak looks like on the Gmail mobile app. (Source: Amtrak) (Large preview)

Now, here is what that same email looks like when viewed through my iPhone’s “Smart Invert” setting:

Amtrak email with inverted colors
What a promotional email from Amtrak looks like in Gmail when colors are inverted. (Source: Amtrak) (Large preview)

The clean design of the original with the white font on the deep blue brand colors is gone. Now, there’s a harsh mix of colors and a hard-to-read Amtrak logo in its place.

You can imagine how this kind of inconsistent and disjointed color display would create an off-putting experience for mobile users.

But what do you expect them to do? If they’re struggling with the glare from their mobile device, Dark Mode (or some other brightness adjustment) will make it easier for them to open and read emails in the first place. Even if it means compromising the appearance of the email you so carefully designed.

One bright spot in all this is that the official “Dark Mode” being rolled out to iPhone (and, hopefully, other smartphones) soon won’t alter the look of HTML emails. Only plain-text messages will be affected.

However, it’s important to still be mindful of how the design choices you make may clash with a surrounding black background. Brightly colored backgrounds, in particular, are likely to clash with the surrounding black of your email app.

How do you solve this issue? Unfortunately, you can’t serve different versions of your email to users based on whether they’re viewing it in Dark Mode or otherwise. You’ll just have to rely on your own tests to confirm that potential views in Dark Mode don’t interfere with your design or message.

In addition to the standard testing you do, set your own smartphone up with Dark Mode (or its hack counterpart). Then, run your test email through the filter and see what happens to the colors. It won’t take long to determine what sort of colors you can and cannot design with for email.

Design the Subject Line

The subject line is the very first thing your email subscribers are going to see, whether it shows up as a push notification or they first encounter it in their inbox. What do you think affects their initial decision to click open an email rather than throw it in the Trash or Spam box immediately? Recognizing the Sender will help, but so will the attractiveness of the subject line.

As for how exactly you go about “designing” a subject line, there are a few things to think about. The first is the length.

Marketo conducted a study across 200+ email campaigns and 2 million emails sent to subscribers. Here is what the test revealed about subject line length:

Marketo subject line length test
Marketo tested over 2 million sent emails to determine the ideal subject line length. (Source: Marketo) (Large preview)

Although the 4-word subject line resulted in the highest open rate, it had a poor showing in terms of clicks. It was actually the 7-word subject line that seemed to have struck the perfect balance with subscribers, leading 15.2% of them to open the email and then 10.8% of them to click on it.

While you should still test this with your own email list, it appears that seven words is the ideal length for a subject line.

Next, you have to think about the buzzwords used in the subject line.

To start, keep this list of Yesware’s Spam Trigger Words out of it:

Yesware list of spam-trigger words
Yesware’s analysis and list of the top spam-trigger words. (Source: Yesware) (Large preview)

If you want to increase the chance the email will be opened, read, clicked on and eventually convert on-site, you have to be savvy about which words will appear within the subject line.

What I’d suggest you do is bookmark CoSchedule’s Email Subject Line Tester tool.

CoSchedule email subject line tester
CoSchedule tests and scores your email subject lines with one click. (Source: CoSchedule) (Large preview)

Here’s an example of how CoSchedule analyzes your subject lines and clues you in to what increases and decreases your open rates:

CoSchedule subject line score
The first part of CoSchedule’s subject line analysis and scoring tool. (Source: CoSchedule) (Large preview)

As you can see, CoSchedule tells you which kinds of words increase open rates as well as those that don’t. Do enough of these subject line tests and you should be able to compile a good set of wording guidelines for your writers and marketers.

Further down, you’ll get more insight into what makes for a strongly worded and designed subject line:

CoSchedule subject line recommendations
The second part of CoSchedule’s subject line assessment and recommendations. (Source: CoSchedule) (Large preview)

CoSchedule will provide recommendations on how to shorten or lengthen the character and word counts based on best practices and results.

Finally, at the very bottom of your subject line test you’ll see this:

CoSchedule email client preview
The final part of CoSchedule’s subject line tester includes an email client preview. (Source: CoSchedule) (Large preview)

This gives you (or, rather, your writer) a chance to see how the subject line will appear within the “design” of an email client. It’s not a problem if the words get cut off on mobile. It’s bound to happen. However, you still want everything that does appear to be appealing enough to click on.

Also, don’t forget about dressing up your subject lines with emoji.

When you think about it, emoji in mobile email subject lines make a lot of sense. Text messaging and social media are ripe with them. It’s only natural to use this fun and truncated form of language in email, too.

Campaign Monitor makes a good point about this:

If you replace words with recognizable emoji, you’ll create shorter subject lines for mobile users. Even if it doesn’t shorten your subject line enough to fit on a mobile screen, it’s still an awesome way to make it stand out from the rest of your subscribers’ cluttered inboxes.

The CoSchedule test will actually score you based on how (or if) you used emoji, too:

CoSchedule Emoji Evaluation
CoSchedule suggests that the use of emoji in subject lines will give you an edge. (Source: CoSchedule) (Large preview)

As you can see, CoSchedule considers this a competitive advantage in marketing.

Even just looking at my own email client, my eye is instantly drawn to the subject line from Sephora which contains a “NEW” emoji:

Sephora subject line with emoji
A Sephora email containing an emoji stands out from others in the inbox. (Source: Sephora) (Large preview)

Just be careful with which emoji you use. For starters, emoji are displayed differently from device to device, so it may not have the same effect on some of your subscribers if it’s a more obscure choice.

There’s also the localization aspect to think about. Not every emoji is perceived the same way around the globe. As Day Translations points out, the fire symbol is one that could cause confusion as some countries interpret it as a literal fire whereas some may view it as a symbol for attraction.

That said, emoji have proven to increase both open rates and read rates of emails. Find the right mobile-friendly emoji to include in your subject line and you could effectively increase the number of subscribers who visit your website as a result.

Wrap-Up

There are so many different kinds of emails that go out from websites:

  • Welcome message
  • Post-purchase transaction email
  • Abandoned cart reminder
  • Promotional news
  • Product featurette
  • New content available
  • Account /rewards points
  • And more.

In other words, there are plenty of ways to get in front of email subscribers.

Just remember that the majority of them will first open your email on mobile. And some will reopen it on mobile over and over again until they’re compelled to click on it or trash it. It’s up to you to design the email in a way that motivates them to visit your website and, consequently, convert.

Further Reading on SmashingMag:

Smashing Editorial (ra, yk, il)


Articles on Smashing Magazine — For Web Designers And Developers

×

Powered by WhatsApp Chat

WhatsApp Chat is free, download and try it now here!

× Chat with Us