The Vision framework was introduced by Apple in 2017 at WWDC as part of iOS 11. Its launch marked a turning point in the evolution of machine vision and image analysis, providing developers with native tools to analyze visual content and perform subsequent processing as needed. In 2017, Vision introduced: Text recognition Face recognition Detection of rectangular shapes Barcode and QR code recognition Since its debut, Apple has continuously enhanced the Vision framework, ensuring it evolves to meet modern requirements. By the end of 2024, with the release of iOS 18, Vision now offers: Improved text recognition accuracy with support for a large number of languages Detection of faces and their features The ability to analyze movements The ability to recognize poses, including the position of hands and key points of the human body Support for tracking objects in video Improved integration with CoreML for working with custom machine-learning models Deep integration with related frameworks, such as AVKit, ARKit With the advent of the Vision framework, developers gained the ability to perform advanced image and video analysis tasks natively, without relying on third-party solutions. These capabilities include scanning documents, recognizing text, identifying faces and poses, detecting duplicate images, and automating various processes that streamline business operations. In this article, we will look at the main scenarios of using Vision with code examples that will help you understand how to work with it, understand that it is not difficult, and start applying it in practice in your applications. VNRequest Vision has an abstract class VNRequest that defines data request structures in Vision, and descendant classes implement specific requests to perform specific tasks with an image. All subclasses inherit the initializer from the VNRequest class. public init(completionHandler: VNRequestCompletionHandler? = nil) Which returns the result of processing the request. It is important to clarify that the result of the request will be returned in the same queue in which the request was sent. Where VNRequestCompletionHandler is a typealias. public typealias VNRequestCompletionHandler = (VNRequest, (any Error)?) -> Void Which returns a VNRequest with the results of the request or an Error if the request was not executed due to some system error, incorrect image, etc. The VNRecognizeTextRequest class from the abstract VNRequest class is designed to handle text recognition requests in images. Example of implementing a request for text recognition: import Vision import UIKit func recognizeText(from image: UIImage) { guard let cgImage = image.cgImage else { return } let request = VNRecognizeTextRequest { request, error in // 1 guard let observations = request.results as? [VNRecognizedTextObservation] else { return } // 2 for observation in observations { if let topCandidate = observation.topCandidates(1).first { print("Recognized text: \(topCandidate.string)") print("Text boundingBox: \(observation.boundingBox)") print("Accuracy: \(topCandidate.confidence)") } } } request.recognitionLevel = .accurate request.usesLanguageCorrection = true let handler = VNImageRequestHandler(cgImage: cgImage, options: [:]) // 3 try? handler.perform([request]) // 3 } Create a VNRecognizeTextRequest for text recognition. Receive the results of the text recognition request as arrays of VNRecognizedTextObservation objects. The VNRecognizedTextObservation object contains: An array of recognized texts (VNRecognizedText().string) Recognition accuracy (VNRecognizedText().confidence) Coordinates of the recognized text on the image (VNRecognizedText().boundingBox) Create a request for image processing, and send a request for text recognition. Example: Recognition of tax identification number and passport number when developing your own SDK for document recognition VNDetectFaceRectanglesRequest This class finds faces in an image and returns their coordinates. Example of implementing a face recognition request: import Vision import UIKit func detectFaces(from image: UIImage) { guard let cgImage = image.cgImage else { return } let request = VNDetectFaceRectanglesRequest { request, error in // 1 guard let results = request.results as? [VNFaceObservation] else { return } // 2 for face in results { print("Face detected: \(face.boundingBox)") } } let handler = VNImageRequestHandler(cgImage: cgImage, options: [:]) // 3 try? handler.perform([request]) // 3 } Create a VNDetectFaceRectanglesRequest for face recognition in an image. Receive the results of the text recognition request as arrays of VNFaceObservation objects. The VNFaceObservation object contains: Coordinates of the recognized face VNFaceObservation().boundingBox. Create a request for image processing and send a request for face recognition. Example: In banks, there is a KYC onboarding where you have to take a photo with your passport; this way, you can confirm that this is the face of a real person. VNDetectBarcodesRequest This class recognizes and reads barcodes and QR codes from an image. Example of implementing a request for recognizing and reading a barcode and QR code: import Vision import UIKit func detectBarcodes(from image: UIImage) { guard let cgImage = image.cgImage else { return } let request = VNDetectBarcodesRequest { request, error in // 1 guard let results = request.results as? [VNBarcodeObservation] else { return } // 2 for qrcode in results { print("qr code was found: \(qrcode.payloadStringValue ?? "not data")") } } let handler = VNImageRequestHandler(cgImage: cgImage, options: [:]) // 3 try? handler.perform([request]) // 3 } Create a VNDetectBarcodesRequest for recognition. Get the results of the VNBarcodeObservation object array request. The VNBarcodeObservation object contains many properties, including: VNFaceObservation().payloadStringValue - the string value of the barcode or QR code. Create a request for image processing, and send a request for face recognition. Example: QR scanner for reading QR codes for payment. We've covered the 3 main types of queries in Vision to help you get started with this powerful tool. The Vision framework was introduced by Apple in 2017 at WWDC as part of iOS 11. Its launch marked a turning point in the evolution of machine vision and image analysis, providing developers with native tools to analyze visual content and perform subsequent processing as needed. In 2017, Vision introduced: Text recognition Face recognition Detection of rectangular shapes Barcode and QR code recognition Text recognition Face recognition Detection of rectangular shapes Barcode and QR code recognition Since its debut, Apple has continuously enhanced the Vision framework, ensuring it evolves to meet modern requirements. By the end of 2024, with the release of iOS 18, Vision now offers: Improved text recognition accuracy with support for a large number of languages Detection of faces and their features The ability to analyze movements The ability to recognize poses, including the position of hands and key points of the human body Support for tracking objects in video Improved integration with CoreML for working with custom machine-learning models Deep integration with related frameworks, such as AVKit, ARKit Improved text recognition accuracy with support for a large number of languages Detection of faces and their features The ability to analyze movements The ability to recognize poses, including the position of hands and key points of the human body Support for tracking objects in video Improved integration with CoreML for working with custom machine-learning models Deep integration with related frameworks, such as AVKit, ARKit With the advent of the Vision framework, developers gained the ability to perform advanced image and video analysis tasks natively, without relying on third-party solutions. These capabilities include scanning documents, recognizing text, identifying faces and poses, detecting duplicate images, and automating various processes that streamline business operations. In this article, we will look at the main scenarios of using Vision with code examples that will help you understand how to work with it, understand that it is not difficult, and start applying it in practice in your applications. VNRequest Vision has an abstract class VNRequest that defines data request structures in Vision, and descendant classes implement specific requests to perform specific tasks with an image. VNRequest All subclasses inherit the initializer from the VNRequest class. VNRequest public init(completionHandler: VNRequestCompletionHandler? = nil) public init(completionHandler: VNRequestCompletionHandler? = nil) Which returns the result of processing the request. It is important to clarify that the result of the request will be returned in the same queue in which the request was sent. Where VNRequestCompletionHandler is a typealias. VNRequestCompletionHandler public typealias VNRequestCompletionHandler = (VNRequest, (any Error)?) -> Void public typealias VNRequestCompletionHandler = (VNRequest, (any Error)?) -> Void Which returns a VNRequest with the results of the request or an Error if the request was not executed due to some system error, incorrect image, etc. VNRequest The VNRecognizeTextRequest class from the abstract VNRequest class is designed to handle text recognition requests in images. VNRecognizeTextRequest VNRequest Example of implementing a request for text recognition: import Vision import UIKit func recognizeText(from image: UIImage) { guard let cgImage = image.cgImage else { return } let request = VNRecognizeTextRequest { request, error in // 1 guard let observations = request.results as? [VNRecognizedTextObservation] else { return } // 2 for observation in observations { if let topCandidate = observation.topCandidates(1).first { print("Recognized text: \(topCandidate.string)") print("Text boundingBox: \(observation.boundingBox)") print("Accuracy: \(topCandidate.confidence)") } } } request.recognitionLevel = .accurate request.usesLanguageCorrection = true let handler = VNImageRequestHandler(cgImage: cgImage, options: [:]) // 3 try? handler.perform([request]) // 3 } import Vision import UIKit func recognizeText(from image: UIImage) { guard let cgImage = image.cgImage else { return } let request = VNRecognizeTextRequest { request, error in // 1 guard let observations = request.results as? [VNRecognizedTextObservation] else { return } // 2 for observation in observations { if let topCandidate = observation.topCandidates(1).first { print("Recognized text: \(topCandidate.string)") print("Text boundingBox: \(observation.boundingBox)") print("Accuracy: \(topCandidate.confidence)") } } } request.recognitionLevel = .accurate request.usesLanguageCorrection = true let handler = VNImageRequestHandler(cgImage: cgImage, options: [:]) // 3 try? handler.perform([request]) // 3 } Create a VNRecognizeTextRequest for text recognition. Receive the results of the text recognition request as arrays of VNRecognizedTextObservation objects. Create a VNRecognizeTextRequest for text recognition. Create a VNRecognizeTextRequest for text recognition. VNRecognizeTextRequest Receive the results of the text recognition request as arrays of VNRecognizedTextObservation objects. Receive the results of the text recognition request as arrays of VNRecognizedTextObservation objects. The VNRecognizedTextObservation object contains: VNRecognizedTextObservation An array of recognized texts (VNRecognizedText().string) Recognition accuracy (VNRecognizedText().confidence) Coordinates of the recognized text on the image (VNRecognizedText().boundingBox) An array of recognized texts (V NRecognizedText().string ) NRecognizedText().string Recognition accuracy ( VNRecognizedText().confidence ) VNRecognizedText().confidence Coordinates of the recognized text on the image ( VNRecognizedText().boundingBox ) VNRecognizedText().boundingBox Create a request for image processing, and send a request for text recognition. Example: Recognition of tax identification number and passport number when developing your own SDK for document recognition Create a request for image processing, and send a request for text recognition. Create a request for image processing, and send a request for text recognition. Example: Recognition of tax identification number and passport number when developing your own SDK for document recognition Example: Recognition of tax identification number and passport number when developing your own SDK for document recognition Example: VNDetectFaceRectanglesRequest VNDetectFaceRectanglesRequest This class finds faces in an image and returns their coordinates. Example of implementing a face recognition request: import Vision import UIKit func detectFaces(from image: UIImage) { guard let cgImage = image.cgImage else { return } let request = VNDetectFaceRectanglesRequest { request, error in // 1 guard let results = request.results as? [VNFaceObservation] else { return } // 2 for face in results { print("Face detected: \(face.boundingBox)") } } let handler = VNImageRequestHandler(cgImage: cgImage, options: [:]) // 3 try? handler.perform([request]) // 3 } import Vision import UIKit func detectFaces(from image: UIImage) { guard let cgImage = image.cgImage else { return } let request = VNDetectFaceRectanglesRequest { request, error in // 1 guard let results = request.results as? [VNFaceObservation] else { return } // 2 for face in results { print("Face detected: \(face.boundingBox)") } } let handler = VNImageRequestHandler(cgImage: cgImage, options: [:]) // 3 try? handler.perform([request]) // 3 } Create a VNDetectFaceRectanglesRequest for face recognition in an image. Receive the results of the text recognition request as arrays of VNFaceObservation objects. Create a VNDetectFaceRectanglesRequest for face recognition in an image. Create a VNDetectFaceRectanglesRequest for face recognition in an image. VNDetectFaceRectanglesRequest Receive the results of the text recognition request as arrays of VNFaceObservation objects. Receive the results of the text recognition request as arrays of VNFaceObservation objects. VNFaceObservation The VNFaceObservation object contains: VNFaceObservation Coordinates of the recognized face VNFaceObservation().boundingBox. Create a request for image processing and send a request for face recognition. Example: In banks, there is a KYC onboarding where you have to take a photo with your passport; this way, you can confirm that this is the face of a real person. Coordinates of the recognized face VNFaceObservation().boundingBox. Coordinates of the recognized face VNFaceObservation().boundingBox. VNFaceObservation().boundingBox. Create a request for image processing and send a request for face recognition. Create a request for image processing and send a request for face recognition. Example: In banks, there is a KYC onboarding where you have to take a photo with your passport; this way, you can confirm that this is the face of a real person. Example: In banks, there is a KYC onboarding where you have to take a photo with your passport; this way, you can confirm that this is the face of a real person. Example: VNDetectBarcodesRequest VNDetectBarcodesRequest This class recognizes and reads barcodes and QR codes from an image. Example of implementing a request for recognizing and reading a barcode and QR code: import Vision import UIKit func detectBarcodes(from image: UIImage) { guard let cgImage = image.cgImage else { return } let request = VNDetectBarcodesRequest { request, error in // 1 guard let results = request.results as? [VNBarcodeObservation] else { return } // 2 for qrcode in results { print("qr code was found: \(qrcode.payloadStringValue ?? "not data")") } } let handler = VNImageRequestHandler(cgImage: cgImage, options: [:]) // 3 try? handler.perform([request]) // 3 } import Vision import UIKit func detectBarcodes(from image: UIImage) { guard let cgImage = image.cgImage else { return } let request = VNDetectBarcodesRequest { request, error in // 1 guard let results = request.results as? [VNBarcodeObservation] else { return } // 2 for qrcode in results { print("qr code was found: \(qrcode.payloadStringValue ?? "not data")") } } let handler = VNImageRequestHandler(cgImage: cgImage, options: [:]) // 3 try? handler.perform([request]) // 3 } Create a VNDetectBarcodesRequest for recognition. Get the results of the VNBarcodeObservation object array request. Create a VNDetectBarcodesRequest for recognition. Create a VNDetectBarcodesRequest for recognition. VNDetectBarcodesRequest Get the results of the VNBarcodeObservation object array request. Get the results of the VNBarcodeObservation object array request. VNBarcodeObservation The VNBarcodeObservation object contains many properties, including: VNBarcodeObservation VNFaceObservation().payloadStringValue - the string value of the barcode or QR code. Create a request for image processing, and send a request for face recognition. Example: QR scanner for reading QR codes for payment. VNFaceObservation().payloadStringValue - the string value of the barcode or QR code. VNFaceObservation().payloadStringValue - the string value of the barcode or QR code. VNFaceObservation().payloadStringValue Create a request for image processing, and send a request for face recognition. Create a request for image processing, and send a request for face recognition. Example: QR scanner for reading QR codes for payment. Example: QR scanner for reading QR codes for payment. Example: We've covered the 3 main types of queries in Vision to help you get started with this powerful tool.
