Transkriptor

transkriptor

As the hype around AI technologies like chatbots, large language models (LLMs), and copilots started after the release of ChatGPT, many more use cases soon appeared. I’m not talking about the most common ones like writing assistants, image generators, or chatbots trained on your data. I’m talking about more niche use cases like realistic talking avatars, video editors, or speech synthesis. In this article, we’re going to explore one of those use cases – a transcriber app that you can use with podcasts or videos. It will be of great value if you’ve ever wanted to use the GPT-3 API or see an OpenAI Whisper demo in action. YouTube For part 1 of this tutorial, we’ll go through everything we need to get set up and start the development of the app. And then , we’ll upload our working code to the cloud and get a user interface working so that others can see the results and interact with the app. in part 2 Now, buckle up because we’ll get our hands dirty with some Python code building this. Setting the stage To build this, we’ll go through 3 acts. In the first one, we’ll see everything we need to get ready for the journey ahead. In the second one, we’ll build a prototype locally and find out what tools can assist us in the process. And in the final one, we’ll upload everything that we built to the cloud so that others can try it as well. Sounds good? Alright then, let’s go for it! Act 1: The Setup (Whisper and GPT-3 API) First things first, a rough overview of the process we will follow is extracting the information we want to transcribe and then feeding it to . OpenAI’s Whisper From there, we’ll take the output and carry out some post-processing on it before displaying it in a front-end application for others to see and be able to interact with it. This process would take a couple of extra steps if we did it manually. Fortunately, OpenAI made available . API access for both Whisper and ChatGPT Here you’ll have to decide which path to take. Building this locally in your machine (assuming you have Python 3 installed) or, Building it online using a . Google Colab notebook Choose your path first and then we’ll proceed. Next, we’ll need an API key from OpenAI to access the GPT-3.5 Turbo model (which is basically the model that powers ChatGPT). If you don’t know how to get one, follow these steps: Go to the and log in or sign up (depending on your case). OpenAI platform website Then, click on your profile icon in the top right to display the options and select “View API keys”. On that page, you’ll see an option that says “create new secret key” under the API keys section. It will display a popup where your secret key is displayed. in a secure place because you won’t see it ever again. Save it Once you have that key, we’ll go for the next piece of the puzzle, Whisper. You can download and install the latest version of Whisper on your machine by running the following command: pip install -U openai-whisper Alternatively, you can install the latest commit from Git with its dependencies using this command: pip install git+https://github.com/openai/whisper.git This gives you more control of what version you are installing (and it’s the command you’ll want to run if you’re trying this from a Colab notebook.) Note: You will need to have ‘ffmpeg’ installed if you want to run Whisper from the command line. With Whisper installed, all we need now are the actual files we want to transcribe. We can feed it several types of media files like ‘mp3’, ‘mp4’, ‘mpeg’, ‘wav’, and many more. The only limitation here is the size of the files, it will take much more time with a 10-minute video than a 20-minute podcast. Now, here’s the next fork in the road. You’ll have to choose if you want to transcribe or . video audio For the sake of simplicity in this example, we’ll go with using audio files from podcasts. And here, the question that could be on your mind is “ ”. Where do I download podcast episodes from? There are 2 ways that I know of. The 1st one is to use a site like ListenNotes, choose any podcast you like, and go to the individual episodes. It’s worth noting that you’ll have to create an account to be able to access the individual episodes and their corresponding audio files. From there, you’ll have to click on a button with 3 dots to open up a “more” tab and then you’ll see the option to download the audio. Here’s how that will look: The 2nd option is to use Podchaser instead. You’ll follow a similar process of finding a podcast, going through the episode list, and choosing a particular one. Once you get to the page of the individual episode, you’ll see a column on the right-hand side that has the play button and other options underneath. If you scroll down, you’ll see info about the podcast, options to share on socials, and a small section with the option to download the audio file. Here’s how that would look: Choose whichever option you prefer and download a couple of episodes that have different lengths so you can test how the transcription process works. My recommendation is that you choose episodes that are shorter than 30 minutes. The reason for this is that if you choose a longer one, Whisper might take a long time to process it and you could run into timeout issues (Especially if you are using the Colab environment). An episode of 20-30 minutes can take from 4 to 8 minutes to get transcribed. The main bottleneck here is the duration of the transcription process before we get a result that we can use for further work. Now that we have fulfilled all the requirements and the setup is completed. It’s time for us to move on to the next part of the journey. Act 2: The Development (OpenAI Whisper Demo) We are getting into deeper programming territory. We’ll use Whisper through since that’s more convenient. And we don’t need to do anything special for the transcription, just pass the audio file to the model and wait for the transcription, that’s it. Python So let’s go through that process line by line. import whisper

model = whisper.load_model("base") Here we load the ‘base’ version of the Whisper model. There are several versions (or “sizes”) that OpenAI released and they vary on the number of parameters, resources they consume, and time it takes to perform the transcription. Here’s a handy chart for reference. You can go ahead and try them all for size (pun intended) if you want. Bear in mind that if you use a bigger model, you’ll need more resources for running it (like a GPU). So, if you’re running this in Colab, you could use this command instead. model = whisper.load_model('medium', device='cuda') You pass in the parameter of the device you want this to be run on. After you’ve chosen the size of the model, you can then pass in the audio file you want to transcribe, like this: result = model.transcribe(“path-to-where-the-file-is”) Then you can print the contents to verify the transcription worked. (But you may want to only peek into the first 1000 characters or so, instead of printing out the entire thing.) podcast_transcript = result[‘text’]
podcast_transcript[:1000] If that turned out well, you now have the results of the transcription process saved to a variable. The one we will use for post-processing. That’s easy enough. But let’s suppose that we want to have a way to know what was discussed in the podcast beyond what we would find in the episode description. We can create a function that takes the whole transcript and gives us an “executive summary”. A summary that we can also share with busy people who don’t have the time to go through the entire thing and see if it’s worth their time or not. Let’s go ahead and do that. Any guesses for what we’ll use to make that summary? Hint: We already have an API key for it. Yep, you got it. We’ll use the `get-3.5-turbo` model to create this summary by providing it with the transcript of the podcast. We could do the calls directly to the API, but it’s much more convenient to use the OpenAI package for it (which is aptly named ‘openai’ 😆). We will also need the tokenizer library from OpenAI to know the amount of tokens we’ll be sending to the model. And we can install both of those by running… pip install openai
pip install tiktoken Now, we need to save the API key somewhere before we make the call to GPT-3. You can save it as an environment variable and then have it ready to use in the code like this. openai.api_key = os.environ["OPENAI"] Or if you’re using the Colab environment, you can do something like this. import openai
from getpass import getpass

openai.api_key = getpass("Enter your key here: ") And then, when you run that command it’ll display an input cell where you can paste the key. Now, you can confirm that the key works by listing the models currently available with this piece of code. models = openai.Model.list()
for model in models['data]:
  print(model['root']) With our key working correctly, we can go ahead and make the call to the GPT-3 API. But there’s something important we have to take into account first. And that’s related to what I mentioned earlier about “tokens”. Since the idea is to pass the podcast transcript to the AI model and get back a summary of it, we need to know how many “tokens” it’s capable of accepting. The AI doesn’t count words or characters but instead uses tokens. That’s why the concept of “context window” exists. If we exceed the length of the context window, our transcript can get truncated and we’ll get a summary from incomplete data. (Another reason for choosing a podcast episode that’s not too long.) Fortunately, there are 2 versions of the GPT-3 model. The traditional one and another with an extended context window. If you check the OpenAI docs, you can see the difference between those. And that’s what the `tiktoken` library is for. It will help us to know how many tokens are in the transcript, that way we know what model to use for the summary. We can make that calculation with the following code. import tiktoken 
enc = tiktoken.encoding_for_model(“gpt-3.5-turbo”)
print(“Number of tokens in the input ”, len(enc.encode(podcast_transcript))) If the amount of tokens in the result is greater than 4,097, then we’ll have to use the ‘16k’ version of the model. After knowing this, we can finally make the call to the API using the proper model for the situation. First, let’s save the prompt we’ll send to the model in a variable. You can experiment with your own prompt, but here’s an example: prompt = """
You are an expert marketer responsible for the growth of several popular podcasts. You are now tasked with summarizing an episode in a way that is concise and covers the main points of the episode for others to understand.

The transcript of the episode is provided below.
"""

request = prompt + podcast_transcript Now, let’s make the call to the API using the following code. chatOutput = openai.ChatCompletion.create(model="gpt-3.5-turbo-16k",
    messages=[{"role": "system", "content": "You are a helpful assistant."},
                        {"role": "user", "content": request}
                        ]
    ) And then let’s check the response from the API. chatOutput.choices[0].messages.content There you have it! A summary of the podcast episode based on the transcript that we generated using Whisper. You can use this same process to extract the main highlights of the episode. Kind of like a way to provide the potential listener with a peek into the conversation. The way to do this is very similar. The only part that changes is the prompt that the model will receive. Instead of asking for a summary, you can ask it to extract the most important points in a list format. The results depend on your creativity with the prompts. Now that we know this works, we can put all the code into a function to run it locally. So, let’s go ahead and create that function. def transcribe_podcast(file_path):

    print("Starting podcast transcription")


    # Load the Whisper model

    import os

    import Whisper

    print("Loading the Whisper model")

    model = whisper.load_model("medium")

    print(model.device)


    # Do the transcription

    print("Starting the transcription")

    result = model.transcribe(file_path)
    

    # Return the transcribed text  

    print("Transcription completed, returning results...")

    return result You can call that function, sending it the path of an audio file and get a transcription in return. Here’s an example of how you would do that. output = transcribe_podcast(‘path_to_audio_file’) And then peek into the contents of the transcript. output[text][:1000] Alright, so far so good. Now that this works locally, it would be great to upload it to the cloud so that you could show it to others and impress friends & family with your l33t h4xx0r coding skills. 😆 But no really, it’s a good choice to deploy it to an online service so that others can try it. And we’re going to do that in part 2. Making a cloud function that can accept requests and getting a quick frontend done to visualize the results of that function (like the summary & highlights of the podcast episodes). To be continued… We’ll leave it at that for now. So far you have the app working locally and you can test it out with different lengths of episodes and see the average length of time that Whisper takes to make the transcription. Thank you for reading this far. And I hope to see you again for Part 2. Don’t forget to so that you don’t miss the coming articles. subscribe to me on Hackernoon

Using OpenAI's Whisper and GPT-3 API to Build and Deploy a Transcriber App – Part 1

একটি ট্রান্সক্রাইবার অ্যাপ তৈরি এবং স্থাপন করতে OpenAI-এর হুইস্পার এবং GPT-3 API ব্যবহার করা - পার্ট 1

Verwendung der Whisper- und GPT-3-API von OpenAI zum Erstellen und Bereitstellen einer Transcriber-App – Teil 1

Uso de la API Whisper y GPT-3 de OpenAI para crear e implementar una aplicación de transcripción - Parte 1

Utilisation de l'API Whisper et GPT-3 d'OpenAI pour créer et déployer une application de transcription – Partie 1

ट्रांसक्राइबर ऐप बनाने और तैनात करने के लिए ओपनएआई के व्हिस्पर और जीपीटी-3 एपीआई का उपयोग करना - भाग 1

OpenAI の Whisper と GPT-3 API を使用した文字起こしアプリの構築と展開 – パート 1

OpenAI의 Whisper 및 GPT-3 API를 사용하여 전사기 앱 구축 및 배포 – 1부

Usando Whisper e GPT-3 API da OpenAI para construir e implantar um aplicativo Transcriber – Parte 1

Использование OpenAI Whisper и API GPT-3 для создания и развертывания приложения-транскриптера – Часть 1

Transcriber Uygulaması Oluşturmak ve Dağıtmak için OpenAI'nin Whisper ve GPT-3 API'sini Kullanma - Bölüm 1

Sử dụng API Whisper và GPT-3 của OpenAI để xây dựng và triển khai ứng dụng Trình ghi dữ liệu – Phần 1

使用 OpenAI 的 Whisper 和 GPT-3 API 构建和部署 Transcriber 应用程序 - 第 1 部分

AccentFold: Enhancing Accent Recognition - Conclusion, Limitations, and References

Big Data in Pharma: What It Is and How It’s Used

Big Data en Pharma: qué es y cómo se utiliza

Le Big Data dans l'industrie pharmaceutique : de quoi s'agit-il et comment est-il utilisé ?

फार्मा में बड़ा डेटा: यह क्या है और इसका उपयोग कैसे किया जाता है

製薬におけるビッグデータ: その概要とその使用方法

Big Data na indústria farmacêutica: o que é e como é usado

Dữ liệu lớn trong dược phẩm: Nó là gì và nó được sử dụng như thế nào

制药中的大数据：它是什么以及如何使用

Arkworks for Marlin Marlin Fractal R1CS Zero-knowledge proof algorithm Marlin is a R1CS based proof system, that, given a coefficient matrix parameter I = (F, n, m, A, B, C) and a set of valid assignments z = (x, w) ∈ F^n, among which x is public information, namely Instance and is private information, namely, witness if Az ∘ Bz = Cz is established, R1CS is established. If we let z_A=Az, z_B = Bz, z_C = Cz the above formula can be transformed into z_A ∘ z_B = z_C. Therefore, if we can prove that there are four vectors z_A, z_B, z_C, z that satisfy R1CS is established. Transition into Polynomial (Efficiency) Prepare Vanish Polynomial Derivative of Vanish Polynomial It’s a non-zero value if and only if Define Polynomial Define polynomial (LDE) ^zA(X), ^zB(X), ^zC(X) ∈ F<|H| + b[X]z^A(X), z^B(X), z^C(X) ∈ F< |H| + b[X] for vectors z_A, z_B, z_C satisfying that the value on the group H is consistent with the vector, where the order of the group are equal to the length of the vector (assuming they are both), that is: H Add b redundant point without exposing any information of w Define polynomial (LDE) for vector z = (x, w) Define a polynomial ^x(X) ∈ F^{<|H_{[<=|x|]}|}[X] for x, satisfying Define a polynomial for w Define a polynomial ^w(X) (LDE) for a vector bar{w}, satisfying Then the polynomial hat{z}(X) is Define Polynomials for Matrices A, B, C (Holographic) In order to reduce the computational complexity for the verifier (see ), we use a special form to represent the matrix. Below is an example using matrix A. paper 5.2.1 Define: Where t_k is the row index of the kth non-zero value of the matrix, maps the index to the computational domain, and ^val(k) is the kth non-zero value of the matrix, which can be divisible by Therefore, a polynomial can be expressed as Linearity Check In order to prove ^z_A(X) = A^z(X), define the polynomial Which should satisfy The relationship among ^z_A(X), A, ^z(X) in the group H is as shown in the following figure Now, we multiply a factor r(α, X) for each element of ^z_A(X) on group H, then in order to ensure balance, we should multiply a factor r(α,X) for the A^z(X) which is as shown in the following figure It can be seen that when the polynomial t(X) traverses the value of group H, the following is satisfied Likewise, we can also derive it from the formula That is, if it can be proved that the accumulation of polynomials q(X) on group H is 0, the linear relationship among ^z_A(X), A, ^z(X) is established. AHP for R1CS Common Given a polynomial Compute polynomial h_0(X) satisfying Generate random polynomials Prover = a. Commit to ^z_A(X), ^z_B(X), ^z_C(X), ^w(X) , h_0(X),  s(X) b. Transcript.write δ1,cm∗ > Prover = > Oracle Generate random numbers α, η_A, η_B, η_C = > Prover - sumcheck-1 Sumcheck for c. Compute polynomials h_1(X)and g_1(X) such that d. Commit to h_1(X), g_1(X) e. Transcript.write cm_h1, cm_g1 = > Oracle Generate random number β1 = a. Compute s(β_1), h_1(β_1), g_1(β_1), ^z_A(β_1), ^z_B(β_1), ^z_C(β_1), ^w(β_1) b. If we send these numbers to the verifier, the verifier needs to compute > Prover - sumcheck-1 Its computational complexity is Ω(|H||K|), therefore, this part of the calculation needs to be executed by the Prover as a proxy. = > Prover -sumcheck-2 Sumcheck for ∑_{X ∈ H} r(α, X) (η_A^A} (X, β_1) + η_B^B(X,β_1) + η_C^C}(X, β_1)) a. Compute polynomials h_2(X) and g_2(X) such that b. Commit to h_2(X), g_2(X) c. Transcript.write δ_2, cm_h_2, cm_g_2 = > Oracle Generate random number β2 = a. Compute h_2(β_2), g_2(β_2) b. If we send these numbers to the verifier, the verifier needs to compute i. v_H(β_2),  r(α, β_2) ii. ^A(β_2, β_1), ^B(β_2, β_1), ^C(β_2, β_1) > Prover - sumcheck-2 Its computational complexity is Ω(|K|), therefore, this part of the calculation needs to be executed by Prover as a proxy. = > Prover - sumcheck-3 Sumcheck for Define the polynomial which can be combined as: a(X) − b(X)(Xg_3(X) + δ3/|K|) = h_3(X)v_K(X) where b. Commit to h_3(x), g_3(x)) c. Transcript.write δ3, cm_h_3, cm_g_3 = > Oracle Generate random number β_3 = > Prover - sumcheck-3 a. Compute b. Send to the verifier Verifier = > Verifier-sumcheck-3 a. Compute b. Verify If it passes the verification, it means that the value computed by the Prover is valid, then go to the previous level for verification. = > Verifier-sumcheck-2 Recall the equality a. Compute i. v_H(β2) b. Verify If it passes the verification, it means that the value computed by the Prover is valid, then go to the previous level for verification. = > Verifier-sumcheck-1 Recall the equality a. Compute i. v_H(β_1) ii. r(α, β1) iii. v_H[≤|x|](β_1) iv. ^z(β_1) = ^w(β_1) v_{H[<=|x|]}(β_1) + ^x(β_1) b. Verify If it passes the verification, it means that the polynomials ^z_A(X), ^z_B(X), ^z_C(X) and ^z(X) satisfy a linear relationship. = > Verifier Verify ^z_A(β_1) . ^z_B(β_1) -  ^z_C(β_1) = h_0(β_1)v_H(β_1) Polynomial Commitment The protocol has carried out three rounds of interaction in total. The polynomials of each round of interaction commitment and the query points are as follows: Sumcheck - 1 Sumcheck - 2 Sumcheck - 3 Extended KZG10 (cross multi-poly) Optimization Sum(s(X)) = 0 Generate random polynomials Then, for sumcheck - 1 Reduce sumcheck According to the optimization mentioned in the paper （Fractal）we make COS20. Claim6.7 For matrices A, B, C the transformed matrices are Define polynomial t(X) Then, for sumcheck - 1，the formula becomes Common Given the polynomial Compute polynomial h_0(X) satisfying Generate random polynomials Prover = > Prover a. Commit to ^z_A(X), ^z_B(X), ^z_C(X), ^w(X) ,h_0(X), s(X) b. Transcript.write δ1,cm∗δ1,cm∗ = > Oracle Generate random number α, η_A, η_B, η_C = > Prover-sumcheck - 1 Sumcheck for a. Compute polynomials h_1(X) and g_1(X) such that: b. Commit to h_1(X), g_1(X) c. Transcript.write cm_h_1, cm_g_1 = > Oracle Generate random number β_1 = > Prover-sumcheck - 1 a. Compute  s(β_1), h_1(β_1), g_1(β_1), ^z_A(β_1), ^zB(β_1), ^zC(β_1), ^w(β_1)* , v*{H[<=|x|]}(β_1), r(α, β_1) ii. ^z(β_1) = ^w(β_1) v{H[<=|x|]}(β_1) + ^x(β_1) iii. t(β_1) = η_AA^ (β_1, α) + η_CC^*(β_1, α) b. i. If we send these numbers to the verifier, the verifier needs to compute i. v_H(β_1) (β_1, α) + η_BB^ Its computational complexity is Ω(|K|), Therefore, this part of the calculation needs to be executed by Prover as a proxy. = > Prover - sumcheck-2 Sumcheck for Define the polynomial a. Compute polynomials h_2(x) and g_2(x) such that which can be combined as a(X) − b(X)(Xg_2(X) + t(β1)/|K|) = h2(X)v_K(X) where b. Commit to h_2(x), g_2(x) c. Transcript.write t(β1), cm_h_2, cm_g_2 = > Oracle Generate random number β_2 = > Prover - sumcheck-2 Compute b. Send them to the verifier Verifier = > Verifier sumcheck - 2 Compute a. v_K(β2) b. 2. Verify If it passes the verification, it means that the calculation t(β1) calculated by the Prover is valid, then it goes up to the previous verification. = > Verifier sumcheck - 1 Recall the equality a. Compute i. v_H(β1) ii. μ(α, β1) iii. v_H[≤|x|](β1) iv. ^z(β_1) = ^w(β_1) v_{H[<=|x|]}(β_1) + ^x(β_1) 3. Verify If it passes the verification, it means that the polynomials ^z_A(X), ^z_B(X), ^z_C(X) satisfy a linear relationship. = > Verifier Verify ^z_A(β_1) . ^z_B(β_1) -  ^z_C(β_1) = h_0(β_1)v_H(β_1) Reduce polynomial numbers for Sumcheck - 2 Compress the current verification of the three matrices into the verification of one matrix, namely Represent this polynomial as a sparse matrix. Matrix polynomials are reduced from 9 to 3. Set b = 1 Set b = 1 Final Protocol Marlin in Arkworks

Going Deep into the Specification for Marlin

I was writing a chat bot where a user interacts with a machine learning powered bot, then I wanted to write a general example application for anybody to use it. In this application, there will not be any intelligence. The bot will simply recite what it heard so that anyone can implement his/her own logic. I used Angular & reactive approach and the main logic is mainly the service.  Demo can be found here . A small catch for now this application will only work on Chrome browsers as it's the only browser supports Web Speech API currently. Since I will be using reactive approach, I created an interface with an optional payload parameter named . All our actions will implement this interface. Components will be subscribing to these actions and react to them. Action Action {
	payload?: ;
} SpeakingStarted Action {} SpeakingEnded Action {} ListeningStarted Action {} ListeningEnded Action {} RecognizedTextAction Action { ( ) {}
} SpeakAction Action { ( ) {}
} export interface any export class implements export class implements export class implements export class implements export class implements constructor payload: public string export class implements constructor payload: public string In the constructor, we will inject because web speech API lives outside of Angular, we need NgZone to bring it into the Angular realm. NgZone ( ) { .window = ( unknown) IWindow; .createSpeaker(); .createListener(); .subscriptions();
	} constructor zone: NgZone private this window as as this this this One challenge is that synthesizer & recognizer are separate objects. When synthesizer is speaking, the recognizer picks up and it goes into a loop. So, we need to pause one or the other. As you can see in the snippet above, we are creating speaker and listener separately and then we will set up subscriptions to prevent issues in between them. For example: We are reacting to action (as shown below), when it's received, we stop listening and when or we start listening again. SpeakingStarted SpeakingEnded ListeningEnded subscriptions() { .getType(SpeakingStarted)
    .pipe( tap( .stopListening()),
      takeUntil( .destroy$)
    )
    .subscribe();

  merge( .getType(SpeakingEnded), .getType(ListeningEnded))
    .pipe(
      filter( ! .isSpeaking), tap( .startListening()),
      takeUntil( .destroy$)
    )
    .subscribe(); .getType(SpeakAction)
    .pipe(
      tap( ._speak(text)),
      takeUntil( .destroy$)
    )
    .subscribe();
} this // tap(()=> console.log('will stop recognition')), => () this this this this => () this // tap(()=> console.log('will start recognition')), => () this this this ( ) => text this this We need a couple observables that application can subscribe to. Speaker & listener will use the observable for dispatch their actions. action$ is a utility method so that consumer can just pass the type of action it wants and gets only those events. getType such as getType(SpeakerStarted) . _voices$ = BehaviorSubject( );
voices$ = ._voices$.asObservable(); _activeVoice$ = BehaviorSubject( );
activeVoice$ = ._activeVoice$.asObservable(); _action$ = Subject<Action>();
action$ = ._action$.asObservable();

getType(action: Action | ) { .action$.pipe(
    filter( i action),
    map( i.payload),
    takeUntil( .destroy$)
  );
} private new null this private new null this private new this any return this ( ) => i instanceof ( ) => i this Based on the API, we are creating an instance and setting up the parameters and attaching functions that we are interested in. and are the functions we need in our case. SpeechSynthesisUtterance onstart onend We are assigning functions so that whenever invoked, it will trigger an action. We are wrapping them in so that actions actually work in Angular. Lastly, we are loading voices. zone.run createSpeaker() { .speaker = SpeechSynthesisUtterance(); .speaker.lang = .language; .speaker.onstart = { .zone.run( { .isSpeaking = ; ._action$.next( SpeakingStarted());
    });
  }; .speaker.onend = { .zone.run( { .isSpeaking = ; ._action$.next( SpeakingEnded());
    });
  }; .loadVoices();
} private this new this this this => () this => () this true this new this => () this => () this false this new this To load voices, we need to add function to object on the window (This is not the speaker object). Similarly, we are emitting an event after receiving voices and remove this function after the first run. The reason is function may be invoked more than once during the lifetime of our service. (We could also check if the voices are changed but we don't need such a feature for now). onvoiceschanged speechSynthesis onvoiceschanged loadVoices() { .window.speechSynthesis.onvoiceschanged = { .zone.run( { voices = .window.speechSynthesis.getVoices(); .voices = voices; ._voices$.next(voices); voice_us = voices.find( { i.name.indexOf( .defaultVoiceName) > ;
      }); .onVoiceSelected(voice_us, );
    }); .window.speechSynthesis.onvoiceschanged = ;
  };
} private this => () this => () const this this this const ( ) => i // console.log(i.name); return this -1 this false // we are removing the function after its called, // as we will not need this to be called any more. this null Similar to speaker, we are instantiating our listener, setting up parameters and actions. Documentation can be found for the parameters. here When listener resulted in some value, it will invoke function with possible results. We call method to get the text out of it and dispatch RecognizedTextAction with the actual recognized text. Any component subscribed to this action can get the actual value without dealing with details. onresult extractText When listener ended, we are restarting the listener so it can start over. Full Service: { Observable, merge, Subject, BehaviorSubject } ; { Injectable, NgZone } ; { map, filter, tap, takeUntil } ; IWindow Window {
	webkitSpeechRecognition: ;
	SpeechRecognition: ;
	SpeechSynthesisUtterance: ;
} RecognizedText {
	term: ;
	confidence: ;
	isFinal: ;
} Action {
	payload?: ;
} SpeakingStarted Action {} SpeakingEnded Action {} ListeningStarted Action {} ListeningEnded Action {} RecognizedTextAction Action { ( ) {}
} SpeakAction Action { ( ) {}
} ({
	providedIn: ,
}) SenseService { defaultVoiceName = ; language = ;

	destroy$ = Subject(); : IWindow;
	listener: ;
	speaker: ;

	isAllowed = ;

	voices: [] = ; _voices$ = BehaviorSubject( );
	voices$ = ._voices$.asObservable(); _activeVoice$ = BehaviorSubject( );
	activeVoice$ = ._activeVoice$.asObservable(); _action$ = Subject<Action>();
	action$ = ._action$.asObservable(); isSpeaking(val: ) { .speaker._isSpeaking = val;
	} isSpeaking(): { !! .speaker._isSpeaking;
	} isListening(val: ) { .listener._isListening = val;
	} isListening(): { !! .listener._isListening;
	} ( ) { .window = ( unknown) IWindow; .createSpeaker(); .createListener(); .subscriptions();
	}

	subscriptions() { .getType(SpeakingStarted)
			.pipe( tap( .stopListening()),
				takeUntil( .destroy$)
			)
			.subscribe();

		merge( .getType(SpeakingEnded), .getType(ListeningEnded))
			.pipe(
				filter( ! .isSpeaking), tap( .startListening()),
				takeUntil( .destroy$)
			)
			.subscribe(); .getType(SpeakAction)
			.pipe(
				tap( ._speak(text)),
				takeUntil( .destroy$)
			)
			.subscribe();
	}

	getType(action: Action | ): Observable< > { .action$.pipe(
			filter( i action),
			map( i.payload),
			takeUntil( .destroy$)
		);
	} createSpeaker() { key = ; ( .window[key]) { .log( ); .speaker = .window[key]; ;
		} .speaker = SpeechSynthesisUtterance(); .window[key] = .speaker; .speaker.lang = .language; .speaker.onstart = { .zone.run( { .isSpeaking = ; ._action$.next( SpeakingStarted());
			});
		}; .speaker.onend = { .zone.run( { .isSpeaking = ; ._action$.next( SpeakingEnded());
			});
		}; .loadVoices();
	} loadVoices() { .window.speechSynthesis.onvoiceschanged = { .zone.run( { voices = .window.speechSynthesis.getVoices(); .voices = voices; ._voices$.next(voices); voice_us = voices.find( { i.name.indexOf( .defaultVoiceName) > ;
				}); .onVoiceSelected(voice_us, );
			}); .window.speechSynthesis.onvoiceschanged = ;
		};
	} createListener() { key = ; ( .window[key]) { .log( ); .listener = .window[key]; .startListening(); ;
		} webkitSpeechRecognition = .window.webkitSpeechRecognition; .listener = webkitSpeechRecognition(); .window[key] = .listener; .listener.continuous = ; .listener.interimResults = ; .listener.lang = .language; .listener.maxAlternatives = ; .listener.maxResults = ; .listener.onstart = { .zone.run( { .isListening = ; ._action$.next( ListeningStarted());
			});
		}; .listener.onresult = { (speech.results) { term: RecognizedText;
				term = .extractText(speech); (term.isFinal) { .zone.run( { ._action$.next( RecognizedTextAction(term.term));
					});
				}
			}
		}; .listener.onerror = { (error.error === ) {
			} (error.error === ) { .isAllowed = ;
			} { .error(error.error);
			}
		}; .listener.onend = { .zone.run( { .isListening = ; ._action$.next( ListeningEnded());
			});
		}; .startListening();
	} stopListening() { .listener.stop();
	} startListening() { (! .startListening) { ;
		} { .log( ); (! .isAllowed) { ;
			} .listener.start();
		} {}
	}

	onVoiceSelected(voice: , speak = ) { .speaker.voice = voice; ._activeVoice$.next(voice); (speak) ._speak( );
	}

	extractText(speech: ): RecognizedText { term = ; result = speech.results[speech.resultIndex]; transcript = result[ ].transcript; confidence = result[ ].confidence; (result.isFinal) { (result[ ].confidence < ) { } {
				term = transcript.trim(); }
		} { (result[ ].confidence > ) {
				term = transcript.trim();
			}
		} <RecognizedText>{
			term,
			confidence,
			isFinal: result.isFinal,
		};
	}

	speak(text: ) { ._action$.next( SpeakAction(text));
	} _speak(text: ): { .log( ); .speaker.text = text; .window.speechSynthesis.speak( .speaker);
	}
} import from 'rxjs' import from '@angular/core' import from 'rxjs/operators' interface extends any any any export interface string number boolean export interface any export class implements export class implements export class implements export class implements export class implements constructor payload: public string export class implements constructor payload: public string @Injectable 'root' export class private 'Google US English' private 'en-US' new window any any true any null private new null this private new null this private new this set boolean this get boolean return this set boolean this get boolean return this constructor zone: NgZone private this window as as this this this this // tap(()=> console.log('will stop recognition')), => () this this this this => () this // tap(()=> console.log('will start recognition')), => () this this this ( ) => text this this any any return this ( ) => i instanceof ( ) => i this private const '_ms_Speaker' if this console 'speaker found' this this return this new this this // this.speaker.voiceURI = 'native'; // this.speaker.volume = 1; // 0 to 1 // this.speaker.rate = 1; // 0.1 to 10 // this.speaker.pitch = 0; //0 to 2 // this.speaker.text = 'Hello World'; this this this => () this => () this true this new this => () this => () this false this new this private this => () this => () const this this this const ( ) => i // console.log(i.name); return this -1 this false // we are removing the function after its called, // as we will not need this to be called any more. this null private const '_ms_Listener' if this console 'recognition found' this this this return const this this new this this this true this true this this this 1 this 25 this => () this => () this true this new this ( ) => speech if let this // console.log(term) if this => () this new this ( ) => error if 'no-speech' else if 'not-allowed' this false else console this => () this => () // console.log('recognition onend'); this false this new this private this private if this return try console 'recognition started' if this return this catch any true this this if this 'Hello' any let '' let let 0 let 0 if if 0 0.3 // console.log('Not recognized'); else // console.log(term); else if 0 0.6 // return term; return string this new private string void console 'speaking...' this this this Usage on any component/service: .senseService.getType(RecognizedTextAction).subscribe( .log(text)); .senseService.speaker( ); .senseService
.getType(RecognizedTextAction)
.pipe(
  debounceTime( ),
  tap( { .senseService.speak( );
  }, takeUntil( .destroy$))
)
.subscribe(); this => text console this 'test speak' // or this 200 ( ) => msg // process ... this `response....` this Demo can be found here .

Speech Recognition And Speech Synthesis on Angular

Chapter 15 from “Reinventing Societal Infrastructure with Technology” which will be released end of January. I will be posting a new section daily. Please share your feedback as this is a work in progress. This is a list of some of the Khosla Ventures’ portfolio companies that are making the biggest difference in the world today. I’ve tried to be thorough with my analysis but I may have missed one or two along the way. For that I apologize. That said, all of these companies are the reason I get out of bed in the morning and the reason I believe we are all moving forward. Health & medicine: AI & data driven change from “practice of medicine to science of medicine” and specialty by specialty reinvention of the expertise, meaning virtual primary care doctors, cardiologists, psychiatrists, oncologists, is happening in our portfolio. Companies like Alivecor enable personal cardiologist-like functions around ECG like diagnosing atrial fibrillation or reading an ECG for normality, although nothing as forward as a full cardiologist yet and definitely not the interventional cardiologist yet; to partially substitute a therapist like Ginger.io, or perhaps the primary care doctor experience like Forward, or create an AI physician like Curai. Others impact health by making accessible, population-scale genetics services like Color, or transforming xenotransplantation into an everyday, lifesaving medical procedure with eGenesis. Genalyte is tackling blood testing results and making it easier to get them in almost real-time in the doctor’s office versus a third-party lab. Rethinking Oncology is the domain of Guardant Health or Oncobox, which is personalizing genetic profiling to help oncologists decide which drug to use. Conquering infectious diseases through the innovative use of next-generation sequencing to analyze microbial cell-free DNA is something Karius does, being able to scan for 1,300 viruses simultaneously. Other companies are using software for hospitals to help doctors and nurses improve key workflows and coordinate a patient’s care through Medisas, or are focused on entirely rebuilding the next-gen health insurance company, like Oscar does. How about anticipating diseases with tools that enable physicians to predict Alzheimer’s disease before the symptoms appear like Neurotrak? For instance, Viome is using predictive biomarkers to your microbiome’s transcriptome and your blood transcriptome at the strain ro RNA level. Using the microbiome, others are creating microbial therapeutics for inflammatory diseases (Siolta) or to target bacteria based on genome (Eligo). Two Pore Guys is building a digital, hand-held, testing platform for DNA and RNA that’s as accurate as medical lab equipment, but is as inexpensive and easy to use as a blood glucose monitor. Zebra is using AI to rethink most radiologist functions for MRI, CT, x-rays for as little as $1, while Vicarious Surgical is enabling surgeons to do much much more with robots. Heartvista is developing a robust commercial MRI system that will give comprehensive cardiovascular evaluation — real-time data with much less skilled technicians! Pharmaceuticals: We believe that there will be AI based reinvention from small molecule chemicals to precise regulation of genes in the pharmaceutical space. Companies are predicting potential drug cures with the use of supercomputers, artificial intelligence, and a specialized algorithm that runs through millions of molecular structures. This can potentially reduce the cost and time involved in making new drug discoveries as our company Atomwise is doing, or combine world-leading expertise in machine learning and genome biology to transform medicine. Such is the case with Deep Genomics who are designing biologicals, or offering low-cost personalized genetic profiling to help cancer doctors decide which drugs will be most effective like Oncobox. Food & food production: AI, plant by plant robotics, and new methods to reducing chemicals, and precision agriculture are changing the way we produce and consume food. From reinventing the burger to be all plant-based to end animal husbandry,while at the same time reducing environmental impact, like Impossible Foods to creating a new plant-based egg substitute using science like Hampton Creek. What if you could develop sustainable and nutritious food products by matching the protein, fat, and carbohydrate distribution and content of traditional dairy products with a mixture of the proteins, fats, and carbohydrates from plants like Ripple? In terms of food production and restaurants, companies are using robotics to make restaurants more efficient, increase food safety, and reallocate funds to higher quality food. On the agricultural process side, imagine using computer vision and robotics to improve agriculture like Blue River (sold to John Deere), who specifically enables farmers to spray herbicides only where weeds are present. Or to make farms more manageable with Granular (sold to Dupont) or climate insurance more accessible with Climate Corporate (sold to Monsanto)? Satellite imaging is allowing farmers to better predict and optimize farming practices like, the services provided by both Rocketlabs and Skybox. Imagine a world where we can use robotics and AI to reduce pesticides, increase efficiency, and create healthier foods — we are only at the beginning. Financial system: Large institutional bodies are often the most posed for disruption from financial services, risk & insurance, blockchain and software contracts-based services, efficient exchanges, accessibility, and even legal services. New small business and consumer services like are provided to by Square. Affirm or Fundbox can make larger purchases more accessible without credit risk, while Stripe can make payment processing easier. Even, as an example, is rethinking insurance and credit for the modern consumer — managing one’s salary, allowing the customer to allocate a small portion per month to an “emergency” amount to be used when a customer may be in a financial bind. Other companies are using blockchain to reimagine aspects of the financial system — for instance, digitizing experimental money to support blockchain to render it more liquid like BlockStream or allowing users to build Bitcoin applications using an enterprise-grade block chain API like Chain. Add adding to liquidity and efficiency are companies like Opendoor which make home selling much easier and faster. Construction & buildings, Housing: 3D-printing, modular factory built, and housing efficiency are a few areas that have opportunity. Imagine 3D-printing a house, or using manufacturing techniques to create lower cost units like Katera, or using robots to transform a small apartment into what feels like a larger space like Ori. Other companies are using robotics to make kitchens and restaurants more efficient like Spyce, and Momentum is changing city spaces by fully roboticized restaurant kitchens. Arevo, as an example, makes ultra-strong 3D-printed carbon fiber composite materials, while Vicarious is creating robots who can help construct and build, replacing human workers. Education: AI & open source reduce need for infrastructure as well as manpower for education. Imagine a platform for academics to share research papers like academia.edu or an online platform that facilitates interaction among students and instructors like Piazza. New platforms from Piazza (college) a and Kiddom for K-12, now used by one in ten of all US teachers, allow for teachers and students to share assignments, provide feedback, grade, and review reports all in one place are enabling more collaboration and better results in education. Manufacturing: 3Dprinting, robotics, new materials, AI design. Imagine 3D-printing metals like Digital Alloys. Velo3D develops a technology that will disrupt 3D-printing in metals and Arevo that is changing composites manufacturing and affordability. Add robotics technologies like Vicarious, Berkshire Grey and manufacturing, which has been relatively isolated from change is feeling the pressures and innovation. Transportation: Imagine rethinking transportation with autonomous cars and ways to help people get where they need to go faster, cheaper, and with less impact on the environment. In the future, we will simply have self-driving cars as part of a fleet of taxis such as Voyage, or perhaps have alternate shorter destination methods of getting around, such as electric skateboards, like Boosted Boards. Hyperloop? Energy & climate, including water: Some companies are developing a completely new high-power, long-cycle life, low-cost battery technology for stationary and select transit applications like Natron, or building a flexible solar cell design that minimizes the amount of semiconducting material used like Caelux. There are breakthroughs in energy conversion such as Quantumscape, a company developing the first commercial solid state electrical energy storage devices for automotive applications. Others are trying to develop and commercialize new battery technology or develop stabilization for electric grids like Varentec. There are companies who are using new techniques and approaches with technology for geothermal energy and Altarock or consuming waste gases form steel and other industrial process to turn that into useful products. Like all great technology, nuclear energy can be used to create new options for converting low-level waste into vast energy sources like Terrapower, while companies like View Glass and Soraa reduce building energy consumption. And, of course, efforts like Breakthrough Energy Ventures have the potential for incubating long-term technologies for decarbonization. **This is a section from “Reinventing Societal Infrastructure with Technology”. To read the previous section, click here.

Khosla Ventures’ Companies Making A Difference Right Now

Stories, News & Mentions about TRANSKRIPTOR

Transkriptor’s Stories on HackerNoon