The grow light meter app to measure PAR / PPFD, DLI, lux / fc, or Kelvin with your iOS or Android phone. Including PPFD meter and DLI conversion for plants.

Photone

The essence of the pattern is that it creates a centralized bus through which components can send and receive messages (events). The bus acts as a mediator between the components, and they can exchange messages without the need for explicit coupling with each other. Hello everyone, my name is Dmitrii Ivashchenko, and I’m a Software Engineer at MY.GAMES. In this article, we’ll talk about an enum-based event bus in Unity, which we’ll get to in just a moment. Back to the event bus pattern itself, there are several ways to implement this, but the general idea remains the same: components send messages (events) to the bus, and other components subscribe to these events and receive them. There are two main methods to implement an event system: and : Subscribe Raise // At the point where the event occurs, we call Raise
// with the event type and data
EventBus.Raise(some_event_type, arguments);

// In the handler, we subscribe to a specific type
// of events and call the OnSomeEventRaised method
EventBus.Subscribe(some_event_type, OnSomeEventRaised); Often the event type is set as a string, but there are also implementations where a class for the passed data is used as the event type: public struct BuyEvent 
{
    public int ItedId;
}

// Raising a new event
EventBus.Raise<BuyEvent>(new BuyEvent{ ItemId = 1 });

// Subscription to event
EventBus.Subscribe<BuyEvent>(OnPlayerBuy); There are also more interesting implementations where interfaces are used for subscription. The event bus can be shared across the entire code, but there may also be separate buses that differ in purpose and method of message delivery. Another interesting way to implement EventBus is Enum-based. Why enums? Using enums as event types has several advantages: Improved code readability Low overhead for Enum processing Code autocompletion in IDE, with ease of adding new types Eliminated possibility of incorrect spelling, which is the case when using strings It adds the possibility of adding extension methods to extend functionality To more easily illustrate the main principle, some details have been omitted in the code examples below. We’ll also be using the class from the library to transmit events. Instead of UniRx, any other means of storing message handlers could also be used, such as C# delegates and events. ReactiveCommand UniRx . The complete source code is available on GitHub Using an Enum as a Event Bus One way to view an is as a type of data bus, with its elements representing concrete types of events delivered by this bus: Enum // Event bus for user interface-generated events
public enum UiEvent
{
    // Event on clicking on the "buy" button 
    BuyButtonClicked,

		// Event on opening an in-game window
		ScreenOpened,

		// Event on completing the tutorial
		TutorialFinished,

    ...
} Using C# extension methods, we can add methods for sending and subscribing to events to this enum: public static class UiEventExtensions
{
    // Declare a static readonly ReactiveCommand of type UiEvent
    private static readonly ReactiveCommand<UiEvent> 
        _uiEventCommand = new ReactiveCommand<UiEvent>();
    
    // Extension method to raise a UiEvent
    public static void Raise(this UiEvent evnt)
    {
        // Execute the ReactiveCommand with the UiEvent
        _uiEventCommand.Execute(evnt);
    }
    
    // Extension method to subscribe to a UiEvent
    public static IDisposable Subscribe(this UiEvent evntType, Action action)
    {
        // Return a subscription to the ReactiveCommand, 
        // filtered by the UiEvent type
        return _uiEventCommand.Where(evnt => evnt == evntType)
                              .Subscribe(evnt => action());
    }
} Now, we can send and receive events in the game logic like so: // Raise the BuyButtonClicked event
**UiEvent.BuyButtonClicked.Raise();**

// Subscribe to the BuyButtonClicked event
var disposable = **UiEvent.BuyButtonClicked.Subscribe**(OnBuyButtonClicked);

// ...

// Unsubscribe from the event
disposable.Dispose(); To pass parameters together with an event, we’ll need an additional class that will store the event type and data. Example implementation: // This class is used to store information about a UI event
public class UiEventParam
{
    // The type of the UI event
    public UiEvent Type { get; }

    // Constructor that sets the type of the UI event
    protected UiEventParam(UiEvent type) => Type = type;

    // Get the data associated with the UI event
    public T Get<T>() => ((UiEventParam<T>) this).Data;
}

// This class is used to store data associated with a UI event
public class UiEventParam<T> : UiEventParam
{
    // The data associated with the UI event
    public readonly T Data;

    // Constructor that sets the type and data of the UI event
    public UiEventParam(UiEvent type, T data) : base(type) => Data = data;
}

public static class UiEventExtensions
{
    // Declare a static readonly ReactiveCommand of type UiEventParam
    private static readonly ReactiveCommand<UiEventParam> _uiEventParam = new ReactiveCommand<UiEventParam>();
    
    // Create a method to raise an event with a generic type
    public static void Raise<T>(this UiEvent evnt, T data) 
        // Execute the ReactiveCommand with a new UiEventParam object containing the event and data
        => _uiEventParam.Execute(new UiEventParam<T>(evnt, data));
    
    // Create a method to subscribe to an event with a generic type
    public static IDisposable Subscribe<T>(this UiEvent evntType, Action<T> action)
    {
        // Return a subscription to the ReactiveCommand, filtering for the specified event type
        return _uiEventParam
            .Where(evnt => evnt.Type == evntType)
            .Subscribe(evnt =>
                // Execute the action with the data from the UiEventParam object
                action(evnt.Get<T>()));
    }

    public static IObservable<T> Where<T>(this UiEvent evntType, Func<T, bool> predicate)
    {
        // Create an observable sequence from the UiEventParam
        var observable = _uiEventParam.Where(evnt =>
            // Check if the event type matches the given event type 
            // and the predicate returns true
            evnt.Type == evntType && predicate(evnt.Get<T>()));
        
        return observable.Select(evnt => evnt.Get<T>());
    }
} In the example above, we created the class for generalized access to the field of the class, and we also added the method that allows filtering events based on the passed data. With these changes, we can further expand the possibilities of using the event bus: UiEventParam Data UiEventParam<T> Where void ExampleOfRaising()
{
    // Raise the SetSoundVolume event with a value
    **UiEvent.SetSoundVolume.Raise(0.5f);**
}

void SubscribeOnEvents()
{
    _disposable = new CompositeDisposable
    {
        // Subscribe to the SetSoundVolume event with any value
        **UiEvent.SetSoundVolume.Subscribe<float>(OnSetVolume),**
        
        // Subscribe to the SetSoundVolume event
        // and call the OnMute method when it is raised with a value of 0f
        **UiEvent.SetSoundVolume.Where<float>(x => x == 0f).Subscribe(OnMute),**
    };
} Encapsulating network events in an enum-based bus Similarly, we can wrap any network interactions in an enum-extension, whether that’s message exchange with the backend, or anything else. Let's consider the example of the network library . We'll write a C# extension that wraps the receiving and sending of events in an enum called : Photon Unity Networking 2 NetEvent public static class NetEventExtensions
{
    // Create a new ReactiveCommand of type NetEventParam
    private static readonly ReactiveCommand<NetEventParam> 
        _netEventParam = new ReactiveCommand<NetEventParam>();

    // Extension method to raise a NetEvent with data
    public static void Raise<T>(this NetEvent evnt, T data)
    {
        // Raise the event with the data, options, and reliable send
        PhotonNetwork.RaiseEvent((byte) evnt, data, GetOptions(evnt), SendOptions.SendReliable);
    }

    // Extension method to subscribe to a NetEvent
    public static IDisposable Subscribe<T>(this NetEvent evntType, Action<T> action)
    {
        // Return a subscription to the ReactiveCommand, 
        // filtered by the event type, 
        // and call the action with the data
        return _netEventParam.Where(evnt => evnt.Type == evntType)
                             .Subscribe(evnt => action(evnt.Get<T>()));
    }
    
    // Method to be called when a PhotonEvent is received
    public static void OnPhotonEventReceived(EventData data)
    {
        // Cast the data code to a NetEvent
        var evnt = (NetEvent) data.Code;
        // Execute the ReactiveCommand with a new NetEventParam containing the event and custom data
        _netEventParam.Execute(new NetEventParam<object>(evnt, data.CustomData));
    }
} The method should be called in your class implementing the . We can now work with network events, just like we did before, with UI events: OnPhotonEventReceived Photon interface IOnEventCallback.OnEvent // Enum for network events
public enum NetEvent
{
    // Event for player state
    PlayerReady,

    // Event for client ping
    ClientPing,

    // Event for battle finish
    BattleFinish,

    // ...
}

// Method to raise the PlayerReady event
public void OnGameLoaded()
{
    // Raise the PlayerReady event with the player's ID
    **NetEvent.PlayerReady.Raise(_playerId);**
}

// Method to subscribe to the PlayerReady event
public void SomeGameManagerOnAnotherDevice()
{
    _disposable = new CompositeDisposable
    {
        // Subscribe to the PlayerReady event
        **NetEvent.PlayerReady.Subscribe<int>(OnPlayerReady)**
    };
} Other types of buses A bus can be characterized not only by the method of delivery, but also by the environment for message propagation. The above examples show buses that deliver events at runtime, but we can also identify a separate bus for editor events (in Edit and Play-mode): [InitializeOnLoad]
public static class EditorBridge
{
    // Static constructor that is called when the class is loaded
    static EditorBridge()
    {
        var disposable = new CompositeDisposable
        {
            // Subscribe to the event and log a message when it is raised
            **EditorEvent.SomeEventName.Subscribe**(() => Debug.Log("Editor event"))
        };

        // Add a handler to the EditorApplication.quitting event
        // that disposes of the CompositeDisposable
        EditorApplication.quitting += () => disposable.Dispose();
    }

    // Create a menu item in the Tools menu
    [MenuItem("Tools/Raise Test Event")]
    public static void RaiseTestEvent()
    {
        **EditorEvent.SomeEventName.Raise();**
    }
} In this case, the implementation of the and methods will not differ from . In Edit-Mode, the event bus can be used to exchange messages between different custom editors, allowing for interface updates and synchronizing data between them. Subscribe Raise UiEventExtensions Another useful type of bus is a . In the previous examples, the sent event is executed synchronously in the same frame. However, there are tasks that require time for processing but do not demand immediate execution. For such events, a separate manager is required which will check whether there is enough time in the current frame to execute the handlers, or whether to wait for the next frame: deferred execution bus public class DelayedEventsProcessor
{
    // This method is called when the game starts
    [RuntimeInitializeOnLoadMethod]
    public static async void Init()
    {
        // Create a cancellation token source
        var source = new CancellationTokenSource();
        // Store the current delta time
        var time = Time.deltaTime;

        // While the cancellation is not requested,
        // the queue is not empty
        // and the execution limit is not exceeded
        while (!source.IsCancellationRequested 
               && !DelayedEventsQueue.IsEmpty()
               && (Time.deltaTime - time) > ExecutionLimit)
        {
            // Process the events in the queue
            DelayedEventsQueue.ProcessEvents();
            // Waiting for next frame
            await Task.Yield();
            // Update the delta time
            time = Time.deltaTime;
        }
        
        // When the application is quitting, cancel the token
        Application.quitting += () => source.Cancel();
    }
} One can separately highlight the bus, and its events will only be handled in development builds and will be turned off in release configurations. DebugEvent Pros and cons of event buses Using the event bus pattern improves Unity game architecture, simplifies development and maintenance, and enhances flexibility and performance. Let’s look in more detail: : The Event Bus pattern allows for the separation of game logic into components, which simplifies its understanding and maintenance. Each component can process only the events it needs, reducing coupling and increasing system flexibility. Separation of logic : Using an event bus makes it easy to add and remove components, simplifying game expansion and functionality changes. Flexibility : Events processed through the bus can be called asynchronously, improving game performance and reducing the likelihood of delays in operation. Asynchronicity : The event bus pattern can be used to create plugins and modules that can be easily integrated into the game. Extensibility However, it should be remembered that the Event Bus can also lead to some negative effects: : events that are processed through the bus can be triggered asynchronously, which makes it difficult to debug the application. Worsening of debugging : if event processing is not organized correctly, unpredictable interaction between components may occur, leading to errors and failures. Loss of flow control : code that uses the event bus may be harder to understand and maintain, as different components may be subscribed to the same event. Increased code complexity : transmitting messages through the bus can increase the delay in application operation, as events must be processed at the other end of the bus. Increased message transmission delay Conclusion We’ve learned how to use enums to mark data buses and message types, and have explored how to wrap event buses transmitted in different ways and in different environments with enums. Putting it all together, we can use this approach to uniformly handle any type of event, thereby improving code readability and comprehension. public void Initialize()
{
    _disposable = new CompositeDisposable
    {
        // Uniform subscription to events 
        // incoming from different buses

        **UiEvent.ExitBattleClick**.Subscribe(OnExitBattleClick),   

        **BackendEvent.BattleReset**.Subscribe(OnBattleReset),

        **DelayedEvent.ChangeWindDirection**.Subscribe<Direction>(OnChangeWindDirection),

        **NetEvent.PlayerReady**.Subscribe<PlayerReadyData>(OnPlayerReady),

        **DebugEvent.KillOpponentUnits**.Subscribe(OnKillOpponentUnits),

        **EditorEvent.SelectScene**.Subscribe<string>(OnEditorSelectScene),
    };
} To recap once more: the event bus pattern has several advantages, such as reducing the coupling between components, more flexible and extensible code, and the ability to handle messages asynchronously. However, its use can lead to code complexity and increase the difficulty of debugging the application. Therefore, before using the pattern, it is necessary to carefully evaluate its necessity and advantages in a specific project. . Source code is available on GitHub

Using an Enum-Based Event Bus Pattern In Unity

Over the past decade, 3D sensors have emerged to become one of the most versatile and ubiquitous types of sensor used in robotics. In many robotic applications, 3D sensing has become the de facto choice for tasks such as near-field object detection and collision avoidance, surface and object inspection, and map creation. There are a number of 3D sensing modalities available. In this roundup, and we’ll be covering the three most common that we work with at Tangram Vision: Stereo CMOS (both active and passive), structured light, and time of flight. We won’t be covering LiDAR, despite the fact that LiDAR data is also three dimensional. LiDAR users, fret not…we’ll be covering LiDAR as its own separate class of sensors in a separate post in the future! 1. Stereo 3D Sensors These ubiquitous sensors bring a combination of good overall performance coupled with low cost. Stereo 3D sensors come in two basic types: passive and active. Structure Core Active Stereo 3D Sensor. Image attribution: , Inc. Occipital Passive stereo 3D sensors are the least expensive of all 3D sensors, as they use off-the-shelf components that are inexpensive to source and manufacture. In addition, they are available in a wide range of baselines, giving users a choice of sensing ranges that are best adapted to their use case. However, most rely on visible light to operate effectively, which means they struggle to perform in low-light or unlit conditions. Active stereo 3D sensors add an infrared (IR) pattern projector which both increases the fidelity of 3D data captured, as well as adding more reliable operation in lower light conditions. However, the IR projectors on these sensors are limited in range, which relegates these sensors to near- and mid-range applications, regardless of baseline. The addition of an IR pattern projector effectively turns these sensors into hybrid stereo/structured light sensors (more on structured light 3D sensors in the next paragraph!). Examples of stereo 3D sensors: — Passive stereo 3D sensors available in large and small form factors, with wide and narrow baselines. The Zed Mini and Zed 2 feature integrated IMUs for kinetic data streams. Stereolabs Zed - Industrial-grade stereo 3D sensors with ruggedized construction and Gigabit Ethernet connections. Ensenso — Active stereo 3D sensor with bespoke IR projector and built-in IMU for kinetic data. Available as an embeddable module or as an encased standalone sensor. Occipital Structure Core — The D series (D415, D435, D435i, D455) are a diverse range of low cost active stereo 3D sensors. The D435i adds a built-in IMU for kinetic data, while the newest D455 adds extended range that is suitable for mid-range robotic navigation. Intel RealSense 2. Structured Light Sensors Structured light is the most ubiquitous 3D sensing modality used in robotics, thanks to the popularity of the original PrimeSense 3D sensors, which included the first generation Microsoft Kinect. Microsoft Kinect for Xbox 360. By Evan Amos — Own work, Public Domain, https://commons.wikimedia.org/w/index.php?curid=16059165 Structured light 3D sensors combine low cost with high-fidelity 3D data capture, as well as performance in a wide set of lighting conditions, with one notable exception: direct sunlight, or bright indirect sunlight. This is because the infrared light source used by structured light sensors gets overpowered by infrared light in the same bandwidth that is naturally present in sunlight. Like stereo 3D sensors, structured light sensors are available in an array of small to large baselines that are suited to tasks that occur at different ranges from the sensor. Examples: — A low-cost structured light 3D sensor that is often used as a drop-in substitute for the original Microsoft Kinect. Orbbec Astra (No longer available new) — The first standalone 3D sensor designed specifically to be used with mobile devices like Apple iPad and iPhone models. Compatible with for depth-powered application development. Now replaced by the active stereo sensor. Occipital Structure Sensor OpenNI Occipital Structure Sensor Mark II (No longer available new) — The most ubiquitous consumer-grade 3D sensor ever produced. Originally launched to support Xbox gaming, the first Kinect became popular with roboticists and hackers for adding robust 3D sensing at low cost to robots, drones and other platforms. Compatible with for depth-powered application development. Microsoft Kinect V1 OpenNI (No longer available new) — The Carmine series is near-identical in specification to the original Microsoft Kinect, and was available in short- and mid-range models. Unlike the original Microsoft Kinect, the Carmine does not include a microphone array. Compatible with for depth-powered application development. PrimeSense Carmine OpenNI — Ruggedized 3D sensors for industrial applications. The Photoneo phoXi line is available in multiple baselines. Photoneo — A range of structured light 3D sensors available in multiple baselines, with USB3 connection for the Zivid One series, and CAT 6A Ethernet connection for the Zivid Two. Zivid 3. Time of Flight Sensors Time of flight (ToF) sensors are effectively LiDAR sensors — aka, light radar. They send out packets of infrared light, and record the time it takes for the infrared light sent out to return. Despite using infrared light sources like structured light sensors, the manner in which ToF sensors transmit and receive light make them less susceptible to interference from bright and indirect sunlight. PMD Time of Flight (ToF) 3D sensor. ToFExpert, , via Wikimedia Commons CC BY-SA 3.0 ToF sensors often have wider range capabilities than structured light sensors, and can operate accurately for longer distance use cases. However, they aren’t able to return 3D images with the fidelity offered by structured light or stereo 3D sensors. Therefore, they are better suited for tasks like navigation assistance versus surface inspections. Examples: — The newest member of the Microsoft Kinect family of 3D sensors is a compact ToF module that adds a 12MP RGB video camera that is registered to the depth stream for high quality color imaging and textures. The Azure Kinect includes a built-in IMU for kinetic data. Microsoft Azure Kinect — Simple, compact ToF 3D sensors that can be purchased as embeddable modules, or as enclosed standalone sensors. PMD Pico — Increasingly hard-to-find. However, like the original Microsoft Kinect, PrimeSense Carmine and original Occipital Structure Sensor 3D sensors, the Xtion 2 is compatible with for depth-powered application development. ASUS Xtion 2 OpenNI Comparing Stereo Sensors to Structured Light to ToF Perception Sensors So why choose one modality over another? Well, we provided hints in the above descriptions for each modality, but let’s get more concrete on why you might choose one over another. At the highest level, the primary factors that determine 3D sensing selection are: Sensor Performance Impact on Host Environmental Limitations Let’s explore each of these factors in detail, and then show how each sensing modality compares. Sensor Performance The main parameters we’ll be exploring here are range, resolution and reliability. Range is perhaps the most important parameter, because different robot tasks are defined by the range in which they occur. For example, mapping environments at high speed for autonomous vehicles is typically left to rotating LiDAR units, as they have ranges that can exceed 100 meters, which allows them to see distant obstacles well before the autonomous vehicle is upon those obstacles. Those same LiDAR units, however, are poorly suited for close visual inspections, as these tasks benefit from sensors with sub-meter ranges that can support enhanced detail capture and accuracy. For stereo and structured light 3D sensors, range is a function of the baseline between the optical elements on the sensor itself. The rule of thumb is that the wider the baseline, the longer the range. Similarly, a narrower baseline creates a shorter range. So why not just spec a 3D sensor with the widest baseline possible? Well, for one, form factor limitations can preclude this approach. But, more importantly, wider baseline sensors’ sensing ranges start further away from the sensor. That means that a wide baseline sensor might ignore objects that are less than two meters from the sensor. A shorter baseline sensor might not have as long of a range, but it might be able to start sensing within .25 meters of the sensor. Therefore, baseline and the resultant range becomes a tradeoff of how near you need to sense versus how distant you need to sense. You get one or the other, but you can’t get both. Most 3D sensors have ranges that start at around 0.5–1m from the sensor, and extend to 3–6m at best. Keep in mind that at the furthest reaches of a sensor’s range, resolution and accuracy can start to diminish rapidly. ToF 3D sensors typically have much greater range than structured light or stereo 3D sensors, and can often receive reliable signals from objects that are up to 25m away. However, ToF sensors often suffer from excessive noise, and resolution can be much lower than similarly priced 3D modalities. Speaking of resolution, this is both a function of range, as well as the nature of the optics used by the sensor. Structured light sensors’ resolution is limited by the density of the pattern that they project (which gets less dense as distance increases), and the resolution of the IR camera that reads the pattern as it falls on objects and environments to be sensed. Stereo sensors’ resolution is limited by the native resolution of the cameras used for the sensor. For tasks such as navigation, high resolution isn’t strictly necessary, unless objects that might cause an obstruction are very fine (an extension cable, for instance, may be on the edge of detectability for some low-resolution 3D sensors). For tasks such as surface inspections and 3D modeling, high resolution is much more desirable. Because these tasks often occur at close range, many structured light and stereo sensors are well suited to this kind of task. ToF sensors are best suited for mid-distance navigation tasks, or skeletal tracking. These tasks don’t require high-resolution sensing, but benefit from the range and operational speed of ToF sensors. Lastly, let’s talk about reliability. For sensors, this comes in two forms: data acquisition reliability, and mechanical reliability. Data acquisition reliability comes down to the inherent limitations of each sensor modality. Stereo sensors, for instance, need sufficient ambient light in the visible spectrum to work properly. Conversely, infrared-based sensors like structured light and ToF suffer when exposed to too much ambient light, causing sensor washout and data integrity issues. These sensors also struggle with black surfaces (they absorb IR wavelengths) and shiny surfaces (they scatter projected IR light). Active stereo sensors with infrared pattern projectors offer a “best of both worlds” approach that broaden sensor reliability over a number of environments and lighting conditions. Mechanical reliability is often the result of compromises inherent to a specific modality, or result from manufacturer cost cutting. Many consumer-grade sensors use USB for data and power transfer; this is fine in a lab or test environment. However, in real-world use, USB can be flaky and error-prone, resulting in intermittent sensor failures and compromised data. Ethernet is much more stable, but is rarely found on consumer-grade sensors. It is much more likely to be found on much more expensive industrial-grade sensors, where it is a prerequisite for a purchase decision. Impact on Host The main parameters we’ll be exploring here are power consumption, compute use, heat dissipation, data transmission and form factor. All robotic hosts have finite amounts of compute and power available. Robots, drones and other vision-enabled devices are often resource balancing acts where multiple sensors, motors, actuators, and other components fight for scarce system resources. Different sensors have different impacts. Sensors with active IR components are much more power hungry than passive stereo 3d sensors, for instance. But passive stereo sensors with high dynamic range (HDR) cameras may use much more compute than a structured light sensor, as the images they capture are much more data rich. Thus, sensor decisions must be made holistically in context of the other components that require system resources (and that subject is complex enough to deserve its own blog post). Similarly, data transmission and storage can often be limited on robots and drones. Again, the example of using an HDR-equipped 3D sensor can strain even robust WiFi networks given the sheer size of the datastreams being captured and transmitted. Lastly, there are industrial design considerations that can limit what sensor options can be specified. These considerations include the size and weight of a sensor (particularly important for lightweight devices like drones), as well as a sensor’s ability to dissipate heat generated. The latter is of particular importance to IR-equipped sensors. The laser-based IR components in these sensors rely on operating within a specific temperature range to maintain the bandwidth at which they operate. Deviating by even a few degrees Fahrenheit outside of the operating temperature range can cause significant variations in sensor readings, corrupting the data generated enough to make it useless. Therefore, considerations must be made to ventilate or insulate these sensors, depending on the environments in which they will be used. Environmental Limitations The main parameters we’ll be exploring here are ambient lighting, and connectivity. We mentioned above that many of these sensors are sensitive to ambient lighting, but it bears repeating. Lighting challenges are one of the most common causes of 3D sensor failures for robots. A robot that worked exceptionally well in a laboratory environment with consistent lighting conditions will suddenly experience drastic and intermittent failures when placed in a real world context with changing, inconsistent lighting conditions. Therefore, it is important to know all the lighting permutations that may exist upon deployment, and account for those during sensor selection. Connectivity is also important. If sensor data is to be processed and used locally, then connectivity isn’t an issue. However, for robots that will be deployed in an environment with low bandwidth, then it is either necessary to compress or filter sensor data on the host before transmitting, or it is necessary to choose a low bandwidth sensor type (for instance, ToF). Again, it is important to test for packet size thresholds under real world bandwidth constraints, as laboratory environments may not sufficiently reveal deficiencies. Synthesizing the Data and Deciding We’ve put together a handy chart showing different , and the 3D sensors that work best for those scenarios. We’ve noted what key decision factors should be considered for each application, and added notes to further clarify why specific choices should be made. robot tasks to be done We hope that you’ve found this guide to 3D sensors useful! Our next sensor roundup will cover LiDAR sensors, so stay tuned. If you’re currently working on a vision-enabled product like a robot or a drone, please take a look at what we do at . We can help you get your product to market faster, and keep it functioning more reliably upon deployment, too. Tangram Vision Also published at https://medium.com/tangram-visions/sensors-101-3d-sensing-66e528f276c4

3 Common Types of 3D Sensors: Stereo, Structured Light, and ToF

This is a tutorial about how we can implement IBM Cloud Storage in our Ruby on Rails 6 projects. We can upload images and manage these assets with Rails. In this tutorial we are going to build a simple app. We can upload a kitty photo and vote for that photo (this tutorial only includes the part of creating a new Rails app, configuring it to save our photo on the cloud, and showing it on an image tag; the design and the part for the photo will be included in the app but is not shown in this tutorial). You can find the app on this repository , and the live version here: . https://github.com/redacuve/my-cat-photos https://my-cat-photos.herokuapp.com/ Before we start, here is a little bit of background. I needed to use file storage for my project on Rails 6. At first Amazon S3 seemed fair to me. There is a lot of documentation about how to use AWS, but there was a problem. For a free account you need a valid credit card. I was looking for other options but they all needed credit cards (Azure, Google Cloud). Then I found IBM Cloud. For a free account you don’t need a credit card. IBM also upgraded their cloud. Now it is compatible with protocol S3 (Simple Storage Service) like Amazon Web Services. Thanks to this protocol you can save objects (they can be files, or in this case images) and every object is saved in a bucket (a container of objects). This is great because the previous cloud of IBM was tricky to use. Creating an account and making our Bucket The first part is, of course, going to the registration URL and registering. We can go to and create our account. https://cloud.ibm.com/registration We need to provide our email and password. Then click on Next. We need to look for our confirmation email. IBM will send us a code and we need to insert that code in the given field and click on Next. We need to add our personal information and click on . Create the account When we create our account, we can read some documentation and we can click on . Then we are going to see the dashboard. Accept I’m not going to explain all of the services that IBM includes, but feel free to check them out. There are some very good services. We can create our resource clicking on “Create resource”. We are going to search “Object Storage” and click on it. There we can see the features of our plan (lite and free). We can add the name of the resource (I will call it my-cat-photos; you can call it whatever you want). And then select “Create”. Now we are on the Cloud Object Storage section. We need to select “Create Bucket”. Here we can customize our bucket clicking on Custom Bucket. But a standard predefined bucket is fine, so we can select that one. it’s important to have a unique bucket name. We can see the location of the bucket (and we can change it) but the default location is OK. We can also see the service credentials, where we can add another one later. We are ready to click on “View Bucket Configuration”. IMPORTANT. Here there is information that we will need later. We need to look at it and save it for later. Specifically, we need: The region Bucket name Endpoint. We can find the first two on bucket details and the last one scrolling a little bit to endpoints and copy the public one. For me, these are the data: us-south Region: my-cat-photos-cos-standard-cpm Bucket name: s3.us-south.cloud-object-storage.appdomain.cloud Endpoint: Now we can obtain the credentials. For that we need to click on the “Service credentials” menu on the left. Here we are going to click on the “New Credential” button. We can name the credential as we want. But we need to click on advanced options. Here we need to toggle on the switch to include the HMAC credential to our credential and click on add. Then our credential will be created. We can expand the credential and copy the values: and access_key_id secret_access_key At this point, we need to have the following: Region Bucket Name Endpoint access_key_id secret_access_key. With this information we can go on to the second part of this tutorial. Creating our app with Rails 6 This is simple. We are going to create the app. It’s straightforward. We need a model for the photo, a controller with 3 methods (index, new, and create), and two views. One for creating the photo and another for displaying all photos (index and new), and of course our model. In this model we need a field of the string type, because here is where we are going to save our image name. We also need to update the routes. So, let’s start. We can open a terminal in Linux (we need rails 6, bundler, and yarn). And hit this command to create our app: $ rails new my-cat-photos Inside the new folder that Rails creates for us, we can create the photos controller. $ rails generate controller photos Then, we can create the model. $ rails generate model Photo title:text description:text votes: image:string float We can start with the model ‘app/models/photo.rb’ working only with the validation of the image. In this case we need to have the title, the description, and the image to be present so we can save the record. validates_presence_of , , < ApplicationRecord class Photo :title :description :image end Then move to  the controller and the methods ‘app/controllers/photos_controller.rb’, it’s simple: @photos = Photo.all @photo = Photo.new @photo = Photo.new(photo_params) @photo.save
      flash[ ] = redirect_to root_path render private params. ( ).permit( , , , , ) < ApplicationController class PhotosController def index end def new end def create if :success 'Photo Created successfully' else :new end end def photo_params require :photo :title :description :votes :image :image_cache end end With the controller done, we can update our routes ‘config/routes.rb’. It only has three routes. The index route is the main route of our project. Rails.application.routes.draw root get , post , do # For details on the DSL available within this file, see https://guides.rubyonrails.org/routing.html 'photos#index' '/upload' to: 'photos#new' '/photos' to: 'photos#create' end Now we can work with the View. First the new view ‘app/views/photos/new.html.erb’ to create the photo. It's important to create the hidden field called _cache at the end. This is so if validation fails, we don't need to search the photo again. That's why this field is included in the “photo_params” function of the controller. We also need to put in the form multipart: true as an option of the HTML because we need to have the file uploaded in our form. <%= %> <% %> <%= %> <% %> <%= %> <% %> <% %> <%= %> <%= %> <%= %> <%= %> <%= %> <%= %> <%= %> <%= %> <%= %> <%= %> <% %> form_with( @photo, , { }) model: local: true html: multipart: true do |form| @photo.errors.any? if < > div < > h2 pluralize(@photo.errors.count, ) "error" </ > h2 < > ul @photo.errors.full_messages.each do |message| < > li message </ > li end </ > ul </ > div end < = > div class "field" form.label :title form.text_field :title </ > div < = > div class "field" form.label :description form.text_field :description </ > div < = > div class "field" form.label :votes form.number_field :votes </ > div < = > div class "field" form.label , :image 'Upload the foto' form.file_field :image </ > div form.hidden_field :image_cache < = > div class "actions" form.submit "Upload Your photo" </ > div end And the index view ‘app/views/photos/index.html.erb’ is simple. It only shows all the fields for every photo. At the end we have our link to the upload route (where there’s the form to upload our image). <% %> <%= %> <%= %> <%= %> <%= %> <% %> <%= %> @photos.each do |photo| < > ul < > li photo.title </ > li < > li photo.description </ > li < > li photo.votes </ > li < > li photo.image </ > li </ > ul end link_to , upload_path "Create new" Here is a simple app (don’t forget to run the migrations). We can create a photo and see all of the photos created, but if we tried to see something similar to an image it’s not going to work. We cannot see any photo, photo name, or URL. Instead in the photo.image field, we can see something similar to this We are going to solve this with the IBM Cloud. For this, we need to use three gems: #<ActionDispatch::Http::UploadedFile:0x00007fb0651ff3c0> Is a gem that simplifies for us the upload of files in ruby applications. We can see the readme here: . Carrierwave: https://github.com/carrierwaveuploader/carrierwave This gem will help us upload the file to the cloud. It uses the S3 protocol. IBM used the same protocol, so we are going to be able to upload and retrieve the images. The readme is here . Fog-aws: https://github.com/fog/fog-aws With this gem we can resize images before uploading, and we are going to use this to upload a thumb version of our main image. Here is the repository . MiniMagick: https://github.com/minimagick/minimagick We can put these gems in our gemfile. gem , gem gem # My Gems # Uploader 'carrierwave' '~> 2.0' # Help with Ibm Cloud (aws configuration) 'fog-aws' # image resizing 'mini_magick' We proceed to install these gems with this command: $ bundle install Configuring Carrierwave and Fow-aws This is the best part of this tutorial. For the configuration of carrierwave, fog-aws, and minimagick, to work with IBM Cloud, we need to have this: Region. Bucket Name. Endpoint. access_key_id. secret_access_key. The first step to do is generate the uploader for the image, the name of the uploader is up to you but I’m going to call it photo. $ rails generate uploader photo This command will generate a file called photo_uploader.rb inside the folder ‘app/uploaders’, this is a configuration file. Here we are going to change some things. Comment the storage :file to storage :fog (to use with our fow-aws gem). Uncomment the extension_whitelist function to only allow uploading images. Uncomment the version :thumb. This will use mini_magick to create a thumb version of our file. To use this gem we need to uncomment this line: ‘include CarrierWave::MiniMagick’. So our file ‘app/uploaders/photo_uploader.rb’ needs to look like this: CarrierWave::MiniMagick
  storage version process [ , ] < CarrierWave::Uploader:: class PhotoUploader Base include :fog def store_dir "uploads/ / / " #{model. .to_s.underscore} class #{mounted_as} #{model.id} end :thumb do resize_to_fit: 50 50 end def extension_whitelist %w(jpg jpeg gif png) end end After we configure our uploader we need to add it to the model. This is simple. We need to mount the uploader to one field on our model. That field needs to be a field of the string type (that's why when we generate our model we made a field called image of the string type) and attach the uploader; in this case, it’s the PhotoUploader. mount_uploader , PhotoUploader
  validates_presence_of , , < ApplicationRecord class Photo :image :title :description :image end Our next step is to configure fog (that line that we uncomment storage :fog). For that we need to create an initializer for fog. This file needs to be called carrierwave.rb and needs to be inside the folder ‘config/initializers’. So we are going to paste this placeholder code in our file ‘config/initializers/carrierwave.rb’. CarrierWave.configure config.fog_credentials = { , , , , , , }
  config.fog_directory  = config.fog_public     = config.fog_attributes = { } do |config| provider: 'AWS' # required aws_access_key_id: 'xxx' # required unless using use_iam_profile aws_secret_access_key: 'yyy' # required unless using use_iam_profile use_iam_profile: true # optional, defaults to false region: 'eu-west-1' # optional, defaults to 'us-east-1' host: 's3.example.com' # optional, defaults to nil endpoint: 'https://s3.example.com:8080' # optional, defaults to nil 'name_of_bucket' # required false # optional, defaults to true cache_control: "public, max-age= " #{ .days.to_i} 365 # optional, defaults to {} end Before configuring this code, we need to store our credentials in a safe place (if we upload the code with the credentials it can be very dangerous because in a public repo everyone can steal the credentials and that is a bad thing). So we are going to use Credentials. Credentials on Rails 6 Using credentials with Rails 6 is so simple. We have a file called credentials.yml.enc inside the folder ‘config’. This file is encrypted with our ‘master.key’ located in the same folder. That's why our master.key file always needs to be ignored for git. One use of the master.key (between many others) is to decrypt our credentials.yml.enc and use the information that is inside (generally used to store credentials of APIs). To access, decrypt, and edit credentials.yml.enc first we need our favorite editor (for example: gedit, emacs, geany, kate, kwrite, vim, mousepad, etc.) and use the name of the editor to edit the information. I’m going to use nano because I love it. We need to run this command in our terminal: $ EDITOR=’nano’ rails credentials:edit Now our favorite editor will display the information of credentials.yml.enc. Here we can simply uncomment these lines: # aws: # access_key_id: 123 # secret_access_key: 345 And add our keys without quotes, so its something like this aws:
  access_key_id: e5e4f3363d2dbd1aebf13630a26078d5
  secret_access_key: b083b1387037b87454298fc1067404b71feec88b5d945d25 We can save our file, and it will be encrypted. Now we can access that data in our Rails app. We just need to call: NameOfTheApp::Application [:keyToFetch] .credentials .nameOfTheCredential If we forget the name of the application we can find it here: ‘config/application.rb’. If I want to access key id for the name of the module (in my case is MyCatPhotos) I need this lines. MyCatPhotos::Application [:access_key_id]

MyCatPhotos::Application [:secret_access_key] .credentials .aws .credentials .aws Finish configuration Now it’s time to configure the initiliazer ‘config/initializers/carrierwave.rb’. We need to put the aws_secret_key_id and aws_secret_access_key from our encrypted credentials. use_iam_profile needs to be false (it’s the default). region is the region of our bucket. host will be nil (it’s the default). endpoint will be our endpoints. These are the parts of fog credentials. The last thing is the config.fog_directory. Here we need to put our bucket name and leave everything with its defaults. So our code needs to look like this: CarrierWave.configure config.fog_credentials = { , MyCatPhotos::Application.credentials.aws[ ], MyCatPhotos::Application.credentials.aws[ ], , , , }
    config.fog_directory  = config.fog_public     = config.fog_attributes = { } do |config| provider: 'AWS' aws_access_key_id: :access_key_id aws_secret_access_key: :secret_access_key use_iam_profile: false region: 'us-south' host: nil endpoint: 'https://s3.us-south.cloud-object-storage.appdomain.cloud' 'my-cat-photos-cos-standard-cpm' false cache_control: "public, max-age= " #{ .days.to_i} 365 end and we are almost done!. Every time we upload an image it’s going to be saved in our cloud. So to retrieve the image and show it on our page we need to access the property URL, and this will be the source of an image tag. In Rails, we can achieve that with <%= image_tag photo.image.url %> So we modify our index view a little bit, ‘app/views/photos/index.html.erb’, to show the image, if it exists. <% %> <%= %> <%= %> <%= %> <%= %> <% %> @photos.each do |photo| < > ul < > li photo.title </ > li < > li photo.description </ > li < > li photo.votes </ > li image_tag photo.image.url !photo.image.url. ? if nil </ > ul end Now our project can show the images that we upload to the cloud. We can also inspect the url of our images and they are created by default with carrierwave and fow. Looks something like this: https://my-cat-photos-cos-standard-cpm.s3.us-south.cloud-object-storage.appdomain.cloud/uploads/photo/image/1/bebudito.jpg?X-Amz-Expires=600&X-Amz-Date=20200623T205500Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=dbd1aebf108e4f33b3630a2e5d5363d2%2F20200623%2Fus-south%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=723299b8ee0e077a7c70265673825873002ed71838ea2ed28e8a9dc2066ad49f If we can retrieve the thumb version it’s with photo.image.thumb.url. This is the end of this tutorial. We left out a little bit of design and functionalities (like the voting). If you like I can make another tutorial about design and functionalities. I didn’t do it here because this tutorial is too long, but you can see the full app on my . repo

How To Upload and Display Images Trough IBM Cloud with Rails 6

It’s surprisingly difficult to know whether an email you’ve received is trustworthy. Why is this so hard? Well, you can’t trust the sender name. You can’t trust the email’s aesthetics. And you can’t trust the copy. To be safe, you need to verify the sender’s domain name and the domain of all outgoing links. definitely Put yourself in the shoes of a random user and try to decipher “efax.hosting.com.mailru.co” or “drive.google.com.download-photo.net”. It’s unreasonable to expect the average user to be on high alert every time they open their inbox, so let’s find a better way. Using color The fundamental problem is that the information you need to determine whether an email is safe is 1) hard to find and 2) hard to understand. In a perfect world, this information would be 1) easy to skim and 2) impossible to misread. There are a handful of ways to verify you’re talking with the right person. You can verify 1) something they are (biometrics), 2) something they have (2FA codes), or 3) something they know (passwords). It’s not immediately clear how we might associate emails with senders’ biometrics or 2FA codes, but we can definitely work off the idea of an “email password”. We provide websites with passwords to verify who we are all the time. What if websites needed to provide a password so that we could verify who they were? And what if these passwords weren’t complicated strings but rather easy-to-skim colors? us https://github.com/turbomaze/colorful-phish Imagine this: the moment you created a password for a website, they created one to use with you. In your welcome email, they told you what this password was: a specific color, unique to your account. From that moment forward, you could rest easy knowing that if an email didn’t contain that exact color, then you were getting phished. Here’s an example of what one of these welcome emails might look like: And we’re done. Colors are simple to implement, easy to skim, and impossible to misunderstand. Check out for a Node.js implementation that will help you eliminate phishing on your site with 3 lines of code. colorful-phish on GitHub If you have any thoughts or want to riff on related ideas, you can find me on Twitter at . https://twitter.com/@imigliu

How Color Can Prevent Your Users from Getting Phished

Enhancing Cross-Chain Communication The goal of Photon is to solve the issues that are caused by interoperability by enabling decentralized apps (dApps) to communicate with one another in a seamless way across many blockchains. This is true independent of whether or not the blockchains in question are based on Ethereum Virtual Machine (EVM) architectures. This objective is accomplished by the company via the provision of high-speed communications services that connect more than 15 blockchains. Furthermore, the company has aspirations to grow this network even further in the year to come. Introduction to Photon's Cross-Chain Messaging A new breakthrough is likely to make it simpler for decentralized applications (dApps) to communicate with one another across various blockchain networks. This is due to the fact that blockchain technology is always experiencing advancements. Recent experiments have shown that the Photon cross-chain message that Entangle is developing, which will be made available along with the mainnet of Entangle, is the most efficient signaling protocol in the Web3 environment. The company's innovative approach to cross-chain communications was what made this significant improvement in communication possible. It is a new feature known as "transmitter Agents" that is at the heart of the Photon protocol's ability to minimize the amount of time it takes for blocks to be created and to make the transaction process move more smoothly. The Need for Speed and Efficiency There are a plethora of independent networks that make up the world of blockchain technology. Each of these networks operates under its own set of restrictions. The concept that the value of a smart contract is directly proportional to the degree of its communication capabilities is the philosophical foundation upon which Photon's design philosophy is founded. It is very necessary to have effective communication throughout the various chains in order to make full advantage of the potential offered by Web3 technologies. Enhanced user experiences and more decentralized application capabilities will be made possible as a result of this occurrence. Technical Insights The Transmitter Agent framework is responsible for the technological foundations of Photon's high-speed messaging. This framework makes use of a distributed network of agents to identify and propose cross-chain communications in a proactive manner. Through collaborative efforts, these agents are able to reach an agreement on the authenticity of messages, which enables them to swiftly finish transactions that include many chains. In order to maintain a flexible infrastructure that can accommodate the ever-changing nature of blockchain activities, Photon has implemented lightweight off-chain components and decreased the amount of time it takes for blocks to be generated. Furthermore, developers are granted the liberty to design their own unique consensus thresholds, which further optimizes the validation process in order to enable complicated processes and provide a wide range of functionality. Photon's strategy for ensuring fast and reliable transfers is predicated on the optimized design of the Transmitter Agents, which are strategically distributed to facilitate rapid event capture and proposal submission. This design choice allows for quicker consensus among Transmitter Agents and timely execution of cross-chain messages. Overview of Entangle With the goal of creating a data infrastructure that is entirely adaptive and interoperable, Entangle is a larger project that wants to attract both Web3 applications and existing organizations. The instant supply of photon is not the only thing that this effort does. Entangle's goal is to provide blockchain-based applications and ecosystems communication channels for smart contracts that are not only cost-effective but also secure, interoperable, and allow quick transmission. This will be accomplished via the use of Photon as the company's flagship protocol. There are a wide range of real-world data sources that might potentially be affected by such a data solution. These include, but are not limited to, financial markets, digital asset exchanges, sports, gaming, and a great lot of other examples. Through the provision of developers with the capacity to access and interact with a broad variety of permitted data, Entangle is marketing itself as a supplier of complete data solutions in the Web3 area. Entangle is making this effort as part of its overall strategy to strengthen its position in the industry. Conclusion Photon offers fast speeds, freedom, and security, which will help Web3 dApps reach more people and work better. Photon wants to speed up the creation of a blockchain network environment that is more connected and flexible. Entangle wants to make cross-chain DeFi easier and better for all users. This shows that it is committed to making decentralized technology better and easier to use. Adding these kinds of unique rules will change blockchain and the digital market as it grows. Don’t forget to like and share the story! Vested Interest Disclosure: This author is an independent contributor publishing via our brand-as-author program. Be it through direct compensation, media partnerships, or networking, the author has a vested interest in the company/ies mentioned in this story. HackerNoon has reviewed the report for quality, but the claims herein belong to the author. #DYOR

Cross-Chain Communication Hits Lightspeed with Entangle's Photon

Machine learning is expensive - it requires high computing capacity. For example, training of the OpenAI’s famous GPT-3 model , and requires an amount of energy equivalent to the and creates a carbon footprint equivalent to traveling 700,000 kilometers by car for a single training session. would cost over $4.5M yearly consumption of 126 Danish homes Photonic processors are one of the most promising solutions to the huge computing capacity need and the extreme carbon footprint because they are fast (as we know, light is the fastest thing in the universe) and really energy efficient. Some companies on the market have production-ready photonic processors. One of my favorites is LightOn because their solution is relatively easy, and the way it works is something like magic (you will see why). can do only one thing, but it can do this one thing fast and efficiently. It can multiply a vector with a giant random matrix. The hardware converts the input to a light pattern. A special waveguide does the random matrix multiplication in an analog way. In the end, a camera converts back the result to a vector. The random matrix is fixed, so the multiplication hardware (the waveguide) here is a passive element. But how can help a random matrix multiplicator to machine learning? LightOn’s OPU (Optical Processing Unit) Multiplication with a fixed giant random matrix sounds meaningless, but don’t listen to your intuition, it is a very meaningful thing. It is a convenient way for dimension reduction called random projection. The random projection has a good property; it nearly preserves the distances because of the . So if you have two vectors in a high dimensional space, the distance of the projected vectors will be nearly the same as their original distance. Johnson–Lindenstrauss lemma This distance preserving property can be used very well in reinforcement learning. In LightOn’s example, they are using their OPU to convert the actual game state to a 32 dimension vector. Here the game state is a screen snapshot that is represented by a 33600 (210x160 pixels) vector—the OPU projects from this 33600 dimension space to a 32 dimension space. The learning method is simple Q-learning. The algorithm calculates the Q value and stores it for the (projected) state. If the Q value is unknown, it uses the Q value of the 9 nearest neighbors and calculates the average of them. Because of the Johnson–Lindenstrauss lemma, the distance of the projected vectors is more or less the same as the original state vectors, so the K nearest neighbors of the projected vector are projected from near original vectors. In this case, OPU is used to reduce the dimensions to create an easier problem (a 32 dimension vector is better than a 33600 dimension) which needs less storage, less computing capacity, or a smaller neural network. The same method can be used in recommender systems or any other cases where dimension reduction can be used, and the OPU does it fast on really high dimensional vectors. You can find the code . here Another use case, when OPU is used to project from a low dimensional space to a high dimensional space. In LightOn’s transfer learning example, OPU is used instead of the linear layers of a neural network. In this case, a pretrained CNN (VGG, ResNet, etc.) is used for feature extraction, but the classification is not done by a neural network. Instead of the dense layers, OPU is projecting the feature vector to a high dimensional space where the samples are linearly separable. So, thanks to OPU a simple linear classifier can be used instead of a neural network. The above 2 examples used the OPU to map the original problem to an easier problem that needs less computing capacity and storage, but can it be used to make the neural network training process easier? The answer is yes. There is a method called in this case the . This method has 2 advantages. We can train the hidden layers parallelly and the error is propagated through a fixed random matrix what can be done efficiently with the OPU. Using random matrices instead of backpropagation sounds like magic, but it works. Direct Feedback Alignment error is propagated through fixed random feedback connections directly from the output layer to each hidden layer Here is a nice YouTube video about . DFA As you saw, a relatively simple photonic hardware can help multiple ways to reduce the huge computing capacity needs of machine learning, and OPU is only one of the photonic hardware on the market. There are other solutions with programable matrices or quantum photonic solutions. It is worth keeping your eyes on this field because machine learning needs more and more computing capacity, and photonic processors are one of the most promising ways to provide this.

How Photonic Processors Will Save Machine Learning

You can apply any design, lighting, or graphics style to your 4K image in real-time using this new machine learning-based approach! If you think this looks interesting, watch the video on this topic and read more about it from the references below 👇 Watch the video References ►Read the full article: https://www.louisbouchard.ai/4k-image-translation-in-real-time/ ►Liang, Jie and Zeng, Hui and Zhang, Lei, (2021), "High-Resolution Photorealistic Image Translation in Real-Time: A Laplacian Pyramid Translation Network", https://export.arxiv.org/pdf/2105.09188.pdf ►Code: https://github.com/csjliang/LPTN Video Transcript 00:00 You've all seen these kinds of pictures where a person's face is "toonified" into an anime 00:05 character. 00:06 Many of you must have seen other kinds of image transformations like this, where an 00:10 image is changed to follow the style of a certain artist. 00:13 Here, an even more challenging task could be something like this, where an image is 00:18 transformed into another season or time of the day. 00:21 What you have not seen yet is the time it takes to produce these results and the actual 00:25 resolutions of the produced pictures. 00:28 This new paper is completely transparent towards this as it attacks exactly this problem. 00:33 Indeed, compared to most approaches, they translate high-definition 4K images, and this 00:38 is done in real-time. 00:39 In this work, they showed their results on season translation, night and day translations, 00:45 and photo retouching, which you've been looking at for the last minute. 00:49 This task is also known as 'image-to-image translation', and all the results you see 00:53 here were produced in 4K. 00:55 Of course, this video is not in 4K, and the images were taken from their paper, so it 01:00 might not look that high-quality here. 01:03 Please look at their paper or try their code if you are not convinced! 01:07 These are the most amazing results of this paper. 01:09 Here, you can see their technique below called LPTN, which stands for Laplacian Pyramid Translation 01:16 Network. 01:17 Look at how much less time it took LPTN to produce the image translations where most 01:22 approaches cannot even do it as this amount of definition is just too computationally 01:27 demanding. 01:28 And yes, this is in seconds. 01:30 They could translate 4K images in not even a tenth of a second using a single regular 01:36 GPU. 01:37 It is faster than all these approaches on 480p image translations! 01:41 And yes, it is not eight times faster, but 80 times faster on average! 01:46 But how is that possible? 01:48 How can they be so much more efficient and still produce amazing and high-quality results? 01:53 This is achieved by optimizing the fact that illumination and color manipulation, which 01:58 relates to the style of an image, is contained in the low-frequency component of an image. 02:03 Whereas the content details, which we want to keep when translating an image into another 02:08 style, can be adaptively refined on high-frequency components. 02:13 This is where it becomes interesting. 02:15 These two components can be divided into two tasks that can be performed simultaneously 02:19 by the GPU. 02:21 Indeed, they split the image into low-resolution and high-resolution components, use a network 02:26 to process the information of the low-frequency or the style of the image, 02:30 and render a final image merging this processed style with the refined high-frequency component, 02:37 which is the details of the image but adapted by a smaller sub-network to fit the new style. 02:43 Thus dodging the unavoidable heavy computation consumption when processing the high-resolution 02:48 components in the whole network. 02:50 This has been a long-standing studied field achieved with a popular technique called Laplacian 02:55 Pyramid. 02:57 The main idea of this Laplacian Pyramid method is to decompose the image into high and low-frequency 03:02 segments and reconstruct it afterward. 03:05 First, we produce an average of the initial image, making it blurry and removing high-frequency 03:11 components. 03:12 This is done using a kernel that passes through the whole image to round batches of pixels 03:17 together. 03:18 For example, if they take a 3 by 3 kernel, it would go through the whole image averaging 03:23 3 by 3 patches removing all unique values. 03:26 They are basically blurring the image by softening the edges. 03:30 Then, the difference between this blurry image and the initial image is saved to use at the 03:35 end of the algorithm to re-introduce the details, which are the high-frequency components. 03:41 This is repeated three times with bigger and bigger averaging kernels producing smaller 03:47 and smaller low-frequency versions of the image having less and less high-frequency 03:52 details. 03:53 If you remember, these low-frequency versions of the image contain information about the 03:57 colors in the image and illumination. 03:59 Indeed, they are basically just a blurred low-quality version of our image, which is 04:03 why the model is so much more efficient. 04:06 This is convenient since they are smaller versions of the image, and this is the exact 04:11 information we are trying to change when translating the image into another style. 04:16 Meaning that using these low-frequency versions is much more computationally efficient than 04:21 using the whole image directly, but they are also focused on the information we want to 04:26 change in the image, which is why the results are so great. 04:30 This lower-quality version of the image can be easily translated using an encoder-decoder, 04:35 just like any other image translation technique we previously mentioned, but since it is done 04:40 on a much lower quality image and a much smaller image, 04:43 it is exponentially faster to process. 04:47 The best thing is that the quality of the results only depends on the initially saved 04:52 high-frequency versions of the image sent as input which is not processed throughout 04:57 the whole network. 04:59 This high-frequency information is simply merged at the end of the processing with the 05:03 low-frequency image to improve the details. 05:06 Basically, it is so much faster because the researchers split the image's information 05:11 in two: low-frequency general information and detailed high-frequency information. 05:17 Then, they send only the computational-friendly part of the image, which is exactly what we 05:23 want to transform, the blurry, low-quality general style of the image, or in other words: 05:29 the low-frequency information. 05:30 Then, only fast and straightforward transformations are done on the high-frequency parts of the 05:36 image to resize them and merge them with the blurry newly-stylized image, 05:42 improving the results by adding details on all edges in the picture. 05:46 And voilà! 05:47 You have your results with a fraction of the time and computational power needed. 05:51 This is brilliant, and the code is publicly available if you would like to try it, which 05:56 is always cool! 05:58 As always, the links to the complete article and references are in the description of the 06:02 video. 06:03 Thank you for watching!

High-Resolution Photorealistic Image Translation in Real Time

Is AI the future of video game design? This video is about a paper called Enhancing Photorealism Enhancement. Enhancing Photorealism Enhancement, or EPE, is an AI that can be applied live to the video game and transform every frame to look much more natural. You can find out more about it below. Watch the video References: ►Read the full article: https://www.louisbouchard.ai/the-future-of-video-game ►Richter, Abu AlHaija, Koltun, (2021), "Enhancing Photorealism Enhancement", https://intel-isl.github.io/PhotorealismEnhancement/ Video Transcript 00:00 What you see here is the gameplay of a very popular video game called GTA 5. 00:04 It looks super realistic, but it is still obvious that this is a video game. 00:09 Now, look at this... 00:10 No, this is not real life. 00:13 It is still the same GTA5 gameplay that went through a new model using artificial intelligence 00:18 to enhance its graphics and make it look more like the real world! 00:23 The researchers from Intel Labs just published this paper called Enhancing Photorealism Enhancement. 00:29 And if you think that this may be "just another GAN," taking a picture of the video game as 00:35 an input and changing it following the style of the natural world, let me change your mind. 00:40 They worked on this model for two years to make it extremely robust. 00:44 It can be applied live to the video game and transform every frame to look much more natural. 00:50 Just imagine the possibilities where you can put a lot less effort into the game graphic, 00:54 make it super stable and complete, then improve the style using this model. 00:59 I think this is a massive breakthrough for video games, and this is just the first paper 01:03 attacking this same task applied specifically to video games! 01:07 I want to ask you a question that you can already answer or wait until the end of the 01:11 video to answer: Do you think this is the future of video games? 01:16 If you want more time to answer, that's perfect, let's get into this technique. 01:21 In general, this task is called image-to-image translation. 01:24 You take an image and transform it into another, often using GANs, as I covered numerous times 01:30 in my previous videos. 01:31 If you want an overview of how a typical GAN architecture works, I invite you to check 01:35 out this video appearing on the top right corner as I won't get into the details of 01:40 how it works here. 01:42 As I said earlier, this model is different than basic image-to-image translation as it 01:47 uses the fact that it is applied to a video game. 01:50 This is of enormous importance here as video games have much more information than a simple 01:55 picture, so why make the task more complicated by achieving realistic transformations using 02:01 only the snapshot as input? 02:03 Instead, they use much more information already available for each image of the game like 02:08 the surface normals, depth information, materials, transparency, lighting, and even a segmentation 02:15 map which tells you what and where the objects are. 02:18 I'm sure you can already see how all this additional information can help with this 02:22 task. 02:23 All these images are sent to a first network called the G-buffer Encoder. 02:29 This G-buffer encoder takes all this information, sends it into a classic convolutional network 02:34 independently to extract and condense all the valuable information from these different 02:40 versions of the initial image. 02:42 This is done using multiple residual blocks, as you can see here, which is basically just 02:47 a convolutional neural network architecture, and more precisely, a ResNet architecture. 02:52 The information is extracted at multiple steps, as you can see. 02:56 This is done to obtain information at different stages of the process. 03:00 Early information is vital in this task because it gives more information regarding the spatial 03:05 location and has the smaller details information. 03:08 In comparison, deeper information is essential to understand the overall image and its style. 03:14 A combination of both early and deep information is thus very powerful when used correctly! 03:20 Then, all this information here, referred to as the G-buffer features, is sent to another 03:25 network with the original image from the game called the rendered image. 03:30 You can see the different colors representing the G-buffer information extracted from different 03:35 scales as we previously saw, with the gray arrow showing the process for the actual image. 03:40 Here again, you can see this as an enhanced version of the same residual blocks as for 03:46 the g-buffer encoder repeated multiple times, but with a little tweak to better adapt the 03:51 G-buffer information before being added to the process. 03:54 This is done using what they refer to as RAD here, which is again residual blocks, convolutions, 04:00 and normalization. 04:02 As I mentioned, this architecture is a bit more complicated than a simple encoder-decoder 04:06 architecture like regular GAN. 04:08 Similarly, the training process is also more elaborated. 04:12 Here, you can see two metrics, the realism score, and the LPIPS score. 04:17 The realism score is basically the GAN section of the training process. 04:22 It compares both a similar real-world image to a game image and compares the real image 04:28 to an enhanced game image. 04:29 Helping the model to learn how to produce a realistic and enhanced version of the game 04:34 image sent. 04:36 Whereas this LPIPS component is a known loss used to retain the structure of the rendered 04:41 image as much as possible. 04:43 This is achieved by giving a score based on the difference between the associated pixels 04:48 of the rendered image versus the enhanced image. 04:51 Penalizing the network when it generates a new image that spatially differs from the 04:56 original image. 04:58 So both these metrics work together to improve the overall results during the training of 05:03 this algorithm. 05:04 Of course, as always, you need a large enough dataset of the real world and of the game 05:10 as it won't generate something that the model has never seen before. 05:14 And now, do you think this kind of model is the future of video games? 05:18 Has your opinion changed after seeing this video? 05:21 As always, the references are linked in the description below, and the full article is 05:25 available on my website louisbouchard.ai with more information. 05:29 Thank you for watching!

Photone

growlightmeter.com

-- employees

Since n.d.

COMPANY RANKING

PHOTONE

EVERGREEN INDEX #7400

Photone WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

Photone

growlightmeter.com

-- employees

Since n.d.

COMPANY RANKING

PHOTONE

EVERGREEN INDEX #7400

RELATED COMPANIESRANK

STORIES ABOUT PHOTONE

Photone WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!