In the recent Meta Connect, Mark Zuckerberg mentioned

I think that voice is going to be a much more natural way of interacting with AI than text.

I completely agree with this and it is also much faster than typing your question out, especially when most AI solutions today have some sort of chat baked into them.

In this blog, we will create an API and simple website that transcribes your recording into text using OpenAI’s Whisper model.

Let’s get started!

API Blueprint

First, we need to create the API. It will contain only one method that we will call transcribe_audio

Let’s install the following modules

pip install fastapi uvicorn python-multipart

The code for the blueprint will look like this:

from fastapi import FastAPI, File, UploadFile
from fastapi.responses import JSONResponse

app = FastAPI()

@app.post("/transcribe-audio/")
async def transcribe_audio(file: UploadFile = File(...)):
    try:
        transcribed_text = "Call Whisper here"

        return JSONResponse(content={"lang": "en", "transcribed_text": transcribed_text})

    except Exception as e:
        return JSONResponse(content={"error": str(e)}, status_code=500)

• A single endpoint called transcribe-audio will return the language and the text transcribed
• If there is an exception, an error is returned.

The AI part

Setup Whisper

I have opted to download the CUDA-specific version of PyTorch along with torchaudio

pip install torch==1.11.0+cu113 torchaudio===0.11.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html

Another thing to note is that if you have an Nvidia GPU, you will need to install CUDA drivers for this application to use your GPU. Otherwise, it will target the CPU for execution and you will have slower results.

Next, we install FASTAPI as well as transformers, accelerate, and numpy<2; The reason we want to use a lower version of numpy is to avoid warnings while running with torch.

pip install transformers accelerate "numpy<2"

Finally, we download Whisper from git. I found this to be the easier method of installing the model:

pip install git+https://github.com/openai/whisper.git

With everything installed, we now set up the whisper model

• Import necessary modules
• Set a temp directory to upload the audio files to
• Check if the current device is a CUDA device
• Load the “medium” whisper model. For a full list of models you can check here

from fastapi import FastAPI, File, UploadFile
from fastapi.responses import JSONResponse

import torch

import whisper

from pathlib import Path
import os

# Directory to save uploaded files
UPLOAD_DIR = Path("./uploaded_audios")
UPLOAD_DIR.mkdir(parents=True, exist_ok=True)

# check if the device is a cuda device, else use CPU
device = "cuda:0" if torch.cuda.is_available() else "cpu"
print(f"Using Device: {device}")

# for a full list of models see https://github.com/openai/whisper?tab=readme-ov-file#available-models-and-languages
model = whisper.load_model("medium", device=device)

app = FastAPI()

Now if you run the application using uvicorn main:app --reload, you will see the model loaded successfully (it may take a while depending on the model chosen). Also, if you have installed CUDA drivers, you will see Using Device: cuda:0 in the logs

Transcription

We will edit the transcribe_audio method to perform the transcription using the model

• Save the uploaded file to the uploads directory created above
• Call the transcribe method on the model
• Extract the text and language from the result and return it in the response
• Finally, delete the file uploaded

@app.post("/transcribe-audio/")
async def transcribe_audio(file: UploadFile = File(...)):
    try:
       # Path to save the file
        file_path = f"{UPLOAD_DIR}/{file.filename}"

        # Save the uploaded audio file to the specified path
        with open(file_path, "wb") as f:
            f.write(await file.read())

        # transcribe the audio using whisper model
        result = model.transcribe(file_path)

        # Extract the transcription text from the result
        transcribed_text = result['text']

        return JSONResponse(content={"lang": result["language"], "transcribed_text": transcribed_text})

    except Exception as e:
        print(e.__cause__)
        return JSONResponse(content={"error": str(e)}, status_code=500)

    finally:
        # Optionally, remove the saved file after transcription
        os.remove(file_path)

Result

Now if you run the API and perform a POST request to /transcribe-audio with an audio file in the form data, you will get the following:

{
    "lang": "en",
    "transcribed_text": " Hello hello 123"
}

The reason why I chose “medium” is that it does a good job of punctuation. In the following example, it adds a comma

{
    "lang": "en",
    "transcribed_text": " Hey hey, this is a test"
}

It can also understand different languages

{
    "lang": "ar",
    "transcribed_text": " مرحبا مرحبا اتكلم عربي"
}

Adding a Simple Frontend Integration

AI is wonderful. I just asked it to create a simple website for me to test this API and it regurgitated a lot of code to help me. Thanks ChatGPT.

You can find the code in on my GitHub, but this is how the final product will look like:

Omitted GIF of a long text:

Me attempting Spanish:

Conclusion and Learnings

Using pre-trained models is very easy to implement if you have the hardware to support it. For me, I had an Intel 14700K CPU and a GTX 1080ti GPU. Even though the GPU is a little bit old, I still got some impressive results. It was a lot faster than I expected; transcribing 30 seconds of audio in about 4-5 seconds.

In the Meta Connect event, Mark Zuckerberg demoed his new AI-powered smart glasses by having a conversation with a Spanish speaker, and the glasses displayed subtitles for each person in their respective language. Pretty cool! This is possible using Whisper, however it is only possible with the large model. Alternatively, we can use external libraries like DeepL to perform the translation for us after we transcribe.

Now, how do we fit this inside glasses? 🤔

A nice project to add this to would be a service that gets installed on your OS that will listen to a macro key press or something similar to fill in a text field with an audio recording on demand. This could be a nice side project 🧐 Whisper medium isn't very large but you can also substitute this with Whisper "turbo" or "tiny". I imagine these can run even on mobile devices. Hah, another side project 😄

If you liked this blog, make sure to like and comment. I will be diving more into running AI models locally to see what are the possibilities.

Say Hi 👋 to me on LinkedIn!