The grow light meter app to measure PAR / PPFD, DLI, lux / fc, or Kelvin with your iOS or Android phone. Including PPFD meter and DLI conversion for plants.

Photone

photone

The essence of the pattern is that it creates a centralized bus through which components can send and receive messages (events). The bus acts as a mediator between the components, and they can exchange messages without the need for explicit coupling with each other. Hello everyone, my name is Dmitrii Ivashchenko, and I’m a Software Engineer at MY.GAMES. In this article, we’ll talk about an enum-based event bus in Unity, which we’ll get to in just a moment. Back to the event bus pattern itself, there are several ways to implement this, but the general idea remains the same: components send messages (events) to the bus, and other components subscribe to these events and receive them. There are two main methods to implement an event system: and : Subscribe Raise // At the point where the event occurs, we call Raise
// with the event type and data
EventBus.Raise(some_event_type, arguments);

// In the handler, we subscribe to a specific type
// of events and call the OnSomeEventRaised method
EventBus.Subscribe(some_event_type, OnSomeEventRaised); Often the event type is set as a string, but there are also implementations where a class for the passed data is used as the event type: public struct BuyEvent 
{
    public int ItedId;
}

// Raising a new event
EventBus.Raise<BuyEvent>(new BuyEvent{ ItemId = 1 });

// Subscription to event
EventBus.Subscribe<BuyEvent>(OnPlayerBuy); There are also more interesting implementations where interfaces are used for subscription. The event bus can be shared across the entire code, but there may also be separate buses that differ in purpose and method of message delivery. Another interesting way to implement EventBus is Enum-based. Why enums? Using enums as event types has several advantages: Improved code readability Low overhead for Enum processing Code autocompletion in IDE, with ease of adding new types Eliminated possibility of incorrect spelling, which is the case when using strings It adds the possibility of adding extension methods to extend functionality To more easily illustrate the main principle, some details have been omitted in the code examples below. We’ll also be using the class from the library to transmit events. Instead of UniRx, any other means of storing message handlers could also be used, such as C# delegates and events. ReactiveCommand UniRx . The complete source code is available on GitHub Using an Enum as a Event Bus One way to view an is as a type of data bus, with its elements representing concrete types of events delivered by this bus: Enum // Event bus for user interface-generated events
public enum UiEvent
{
    // Event on clicking on the "buy" button 
    BuyButtonClicked,

		// Event on opening an in-game window
		ScreenOpened,

		// Event on completing the tutorial
		TutorialFinished,

    ...
} Using C# extension methods, we can add methods for sending and subscribing to events to this enum: public static class UiEventExtensions
{
    // Declare a static readonly ReactiveCommand of type UiEvent
    private static readonly ReactiveCommand<UiEvent> 
        _uiEventCommand = new ReactiveCommand<UiEvent>();
    
    // Extension method to raise a UiEvent
    public static void Raise(this UiEvent evnt)
    {
        // Execute the ReactiveCommand with the UiEvent
        _uiEventCommand.Execute(evnt);
    }
    
    // Extension method to subscribe to a UiEvent
    public static IDisposable Subscribe(this UiEvent evntType, Action action)
    {
        // Return a subscription to the ReactiveCommand, 
        // filtered by the UiEvent type
        return _uiEventCommand.Where(evnt => evnt == evntType)
                              .Subscribe(evnt => action());
    }
} Now, we can send and receive events in the game logic like so: // Raise the BuyButtonClicked event
**UiEvent.BuyButtonClicked.Raise();**

// Subscribe to the BuyButtonClicked event
var disposable = **UiEvent.BuyButtonClicked.Subscribe**(OnBuyButtonClicked);

// ...

// Unsubscribe from the event
disposable.Dispose(); To pass parameters together with an event, we’ll need an additional class that will store the event type and data. Example implementation: // This class is used to store information about a UI event
public class UiEventParam
{
    // The type of the UI event
    public UiEvent Type { get; }

    // Constructor that sets the type of the UI event
    protected UiEventParam(UiEvent type) => Type = type;

    // Get the data associated with the UI event
    public T Get<T>() => ((UiEventParam<T>) this).Data;
}

// This class is used to store data associated with a UI event
public class UiEventParam<T> : UiEventParam
{
    // The data associated with the UI event
    public readonly T Data;

    // Constructor that sets the type and data of the UI event
    public UiEventParam(UiEvent type, T data) : base(type) => Data = data;
}

public static class UiEventExtensions
{
    // Declare a static readonly ReactiveCommand of type UiEventParam
    private static readonly ReactiveCommand<UiEventParam> _uiEventParam = new ReactiveCommand<UiEventParam>();
    
    // Create a method to raise an event with a generic type
    public static void Raise<T>(this UiEvent evnt, T data) 
        // Execute the ReactiveCommand with a new UiEventParam object containing the event and data
        => _uiEventParam.Execute(new UiEventParam<T>(evnt, data));
    
    // Create a method to subscribe to an event with a generic type
    public static IDisposable Subscribe<T>(this UiEvent evntType, Action<T> action)
    {
        // Return a subscription to the ReactiveCommand, filtering for the specified event type
        return _uiEventParam
            .Where(evnt => evnt.Type == evntType)
            .Subscribe(evnt =>
                // Execute the action with the data from the UiEventParam object
                action(evnt.Get<T>()));
    }

    public static IObservable<T> Where<T>(this UiEvent evntType, Func<T, bool> predicate)
    {
        // Create an observable sequence from the UiEventParam
        var observable = _uiEventParam.Where(evnt =>
            // Check if the event type matches the given event type 
            // and the predicate returns true
            evnt.Type == evntType && predicate(evnt.Get<T>()));
        
        return observable.Select(evnt => evnt.Get<T>());
    }
} In the example above, we created the class for generalized access to the field of the class, and we also added the method that allows filtering events based on the passed data. With these changes, we can further expand the possibilities of using the event bus: UiEventParam Data UiEventParam<T> Where void ExampleOfRaising()
{
    // Raise the SetSoundVolume event with a value
    **UiEvent.SetSoundVolume.Raise(0.5f);**
}

void SubscribeOnEvents()
{
    _disposable = new CompositeDisposable
    {
        // Subscribe to the SetSoundVolume event with any value
        **UiEvent.SetSoundVolume.Subscribe<float>(OnSetVolume),**
        
        // Subscribe to the SetSoundVolume event
        // and call the OnMute method when it is raised with a value of 0f
        **UiEvent.SetSoundVolume.Where<float>(x => x == 0f).Subscribe(OnMute),**
    };
} Encapsulating network events in an enum-based bus Similarly, we can wrap any network interactions in an enum-extension, whether that’s message exchange with the backend, or anything else. Let's consider the example of the network library . We'll write a C# extension that wraps the receiving and sending of events in an enum called : Photon Unity Networking 2 NetEvent public static class NetEventExtensions
{
    // Create a new ReactiveCommand of type NetEventParam
    private static readonly ReactiveCommand<NetEventParam> 
        _netEventParam = new ReactiveCommand<NetEventParam>();

    // Extension method to raise a NetEvent with data
    public static void Raise<T>(this NetEvent evnt, T data)
    {
        // Raise the event with the data, options, and reliable send
        PhotonNetwork.RaiseEvent((byte) evnt, data, GetOptions(evnt), SendOptions.SendReliable);
    }

    // Extension method to subscribe to a NetEvent
    public static IDisposable Subscribe<T>(this NetEvent evntType, Action<T> action)
    {
        // Return a subscription to the ReactiveCommand, 
        // filtered by the event type, 
        // and call the action with the data
        return _netEventParam.Where(evnt => evnt.Type == evntType)
                             .Subscribe(evnt => action(evnt.Get<T>()));
    }
    
    // Method to be called when a PhotonEvent is received
    public static void OnPhotonEventReceived(EventData data)
    {
        // Cast the data code to a NetEvent
        var evnt = (NetEvent) data.Code;
        // Execute the ReactiveCommand with a new NetEventParam containing the event and custom data
        _netEventParam.Execute(new NetEventParam<object>(evnt, data.CustomData));
    }
} The method should be called in your class implementing the . We can now work with network events, just like we did before, with UI events: OnPhotonEventReceived Photon interface IOnEventCallback.OnEvent // Enum for network events
public enum NetEvent
{
    // Event for player state
    PlayerReady,

    // Event for client ping
    ClientPing,

    // Event for battle finish
    BattleFinish,

    // ...
}

// Method to raise the PlayerReady event
public void OnGameLoaded()
{
    // Raise the PlayerReady event with the player's ID
    **NetEvent.PlayerReady.Raise(_playerId);**
}

// Method to subscribe to the PlayerReady event
public void SomeGameManagerOnAnotherDevice()
{
    _disposable = new CompositeDisposable
    {
        // Subscribe to the PlayerReady event
        **NetEvent.PlayerReady.Subscribe<int>(OnPlayerReady)**
    };
} Other types of buses A bus can be characterized not only by the method of delivery, but also by the environment for message propagation. The above examples show buses that deliver events at runtime, but we can also identify a separate bus for editor events (in Edit and Play-mode): [InitializeOnLoad]
public static class EditorBridge
{
    // Static constructor that is called when the class is loaded
    static EditorBridge()
    {
        var disposable = new CompositeDisposable
        {
            // Subscribe to the event and log a message when it is raised
            **EditorEvent.SomeEventName.Subscribe**(() => Debug.Log("Editor event"))
        };

        // Add a handler to the EditorApplication.quitting event
        // that disposes of the CompositeDisposable
        EditorApplication.quitting += () => disposable.Dispose();
    }

    // Create a menu item in the Tools menu
    [MenuItem("Tools/Raise Test Event")]
    public static void RaiseTestEvent()
    {
        **EditorEvent.SomeEventName.Raise();**
    }
} In this case, the implementation of the and methods will not differ from . In Edit-Mode, the event bus can be used to exchange messages between different custom editors, allowing for interface updates and synchronizing data between them. Subscribe Raise UiEventExtensions Another useful type of bus is a . In the previous examples, the sent event is executed synchronously in the same frame. However, there are tasks that require time for processing but do not demand immediate execution. For such events, a separate manager is required which will check whether there is enough time in the current frame to execute the handlers, or whether to wait for the next frame: deferred execution bus public class DelayedEventsProcessor
{
    // This method is called when the game starts
    [RuntimeInitializeOnLoadMethod]
    public static async void Init()
    {
        // Create a cancellation token source
        var source = new CancellationTokenSource();
        // Store the current delta time
        var time = Time.deltaTime;

        // While the cancellation is not requested,
        // the queue is not empty
        // and the execution limit is not exceeded
        while (!source.IsCancellationRequested 
               && !DelayedEventsQueue.IsEmpty()
               && (Time.deltaTime - time) > ExecutionLimit)
        {
            // Process the events in the queue
            DelayedEventsQueue.ProcessEvents();
            // Waiting for next frame
            await Task.Yield();
            // Update the delta time
            time = Time.deltaTime;
        }
        
        // When the application is quitting, cancel the token
        Application.quitting += () => source.Cancel();
    }
} One can separately highlight the bus, and its events will only be handled in development builds and will be turned off in release configurations. DebugEvent Pros and cons of event buses Using the event bus pattern improves Unity game architecture, simplifies development and maintenance, and enhances flexibility and performance. Let’s look in more detail: : The Event Bus pattern allows for the separation of game logic into components, which simplifies its understanding and maintenance. Each component can process only the events it needs, reducing coupling and increasing system flexibility. Separation of logic : Using an event bus makes it easy to add and remove components, simplifying game expansion and functionality changes. Flexibility : Events processed through the bus can be called asynchronously, improving game performance and reducing the likelihood of delays in operation. Asynchronicity : The event bus pattern can be used to create plugins and modules that can be easily integrated into the game. Extensibility However, it should be remembered that the Event Bus can also lead to some negative effects: : events that are processed through the bus can be triggered asynchronously, which makes it difficult to debug the application. Worsening of debugging : if event processing is not organized correctly, unpredictable interaction between components may occur, leading to errors and failures. Loss of flow control : code that uses the event bus may be harder to understand and maintain, as different components may be subscribed to the same event. Increased code complexity : transmitting messages through the bus can increase the delay in application operation, as events must be processed at the other end of the bus. Increased message transmission delay Conclusion We’ve learned how to use enums to mark data buses and message types, and have explored how to wrap event buses transmitted in different ways and in different environments with enums. Putting it all together, we can use this approach to uniformly handle any type of event, thereby improving code readability and comprehension. public void Initialize()
{
    _disposable = new CompositeDisposable
    {
        // Uniform subscription to events 
        // incoming from different buses

        **UiEvent.ExitBattleClick**.Subscribe(OnExitBattleClick),   

        **BackendEvent.BattleReset**.Subscribe(OnBattleReset),

        **DelayedEvent.ChangeWindDirection**.Subscribe<Direction>(OnChangeWindDirection),

        **NetEvent.PlayerReady**.Subscribe<PlayerReadyData>(OnPlayerReady),

        **DebugEvent.KillOpponentUnits**.Subscribe(OnKillOpponentUnits),

        **EditorEvent.SelectScene**.Subscribe<string>(OnEditorSelectScene),
    };
} To recap once more: the event bus pattern has several advantages, such as reducing the coupling between components, more flexible and extensible code, and the ability to handle messages asynchronously. However, its use can lead to code complexity and increase the difficulty of debugging the application. Therefore, before using the pattern, it is necessary to carefully evaluate its necessity and advantages in a specific project. . Source code is available on GitHub

Using an Enum-Based Event Bus Pattern In Unity

Over the past decade, 3D sensors have emerged to become one of the most versatile and ubiquitous types of sensor used in robotics. In many robotic applications, 3D sensing has become the de facto choice for tasks such as near-field object detection and collision avoidance, surface and object inspection, and map creation. There are a number of 3D sensing modalities available. In this roundup, and we’ll be covering the three most common that we work with at Tangram Vision: Stereo CMOS (both active and passive), structured light, and time of flight. We won’t be covering LiDAR, despite the fact that LiDAR data is also three dimensional. LiDAR users, fret not…we’ll be covering LiDAR as its own separate class of sensors in a separate post in the future! 1. Stereo 3D Sensors These ubiquitous sensors bring a combination of good overall performance coupled with low cost. Stereo 3D sensors come in two basic types: passive and active. Structure Core Active Stereo 3D Sensor. Image attribution: , Inc. Occipital Passive stereo 3D sensors are the least expensive of all 3D sensors, as they use off-the-shelf components that are inexpensive to source and manufacture. In addition, they are available in a wide range of baselines, giving users a choice of sensing ranges that are best adapted to their use case. However, most rely on visible light to operate effectively, which means they struggle to perform in low-light or unlit conditions. Active stereo 3D sensors add an infrared (IR) pattern projector which both increases the fidelity of 3D data captured, as well as adding more reliable operation in lower light conditions. However, the IR projectors on these sensors are limited in range, which relegates these sensors to near- and mid-range applications, regardless of baseline. The addition of an IR pattern projector effectively turns these sensors into hybrid stereo/structured light sensors (more on structured light 3D sensors in the next paragraph!). Examples of stereo 3D sensors: — Passive stereo 3D sensors available in large and small form factors, with wide and narrow baselines. The Zed Mini and Zed 2 feature integrated IMUs for kinetic data streams. Stereolabs Zed - Industrial-grade stereo 3D sensors with ruggedized construction and Gigabit Ethernet connections. Ensenso — Active stereo 3D sensor with bespoke IR projector and built-in IMU for kinetic data. Available as an embeddable module or as an encased standalone sensor. Occipital Structure Core — The D series (D415, D435, D435i, D455) are a diverse range of low cost active stereo 3D sensors. The D435i adds a built-in IMU for kinetic data, while the newest D455 adds extended range that is suitable for mid-range robotic navigation. Intel RealSense 2. Structured Light Sensors Structured light is the most ubiquitous 3D sensing modality used in robotics, thanks to the popularity of the original PrimeSense 3D sensors, which included the first generation Microsoft Kinect. Microsoft Kinect for Xbox 360. By Evan Amos — Own work, Public Domain, https://commons.wikimedia.org/w/index.php?curid=16059165 Structured light 3D sensors combine low cost with high-fidelity 3D data capture, as well as performance in a wide set of lighting conditions, with one notable exception: direct sunlight, or bright indirect sunlight. This is because the infrared light source used by structured light sensors gets overpowered by infrared light in the same bandwidth that is naturally present in sunlight. Like stereo 3D sensors, structured light sensors are available in an array of small to large baselines that are suited to tasks that occur at different ranges from the sensor. Examples: — A low-cost structured light 3D sensor that is often used as a drop-in substitute for the original Microsoft Kinect. Orbbec Astra (No longer available new) — The first standalone 3D sensor designed specifically to be used with mobile devices like Apple iPad and iPhone models. Compatible with for depth-powered application development. Now replaced by the active stereo sensor. Occipital Structure Sensor OpenNI Occipital Structure Sensor Mark II (No longer available new) — The most ubiquitous consumer-grade 3D sensor ever produced. Originally launched to support Xbox gaming, the first Kinect became popular with roboticists and hackers for adding robust 3D sensing at low cost to robots, drones and other platforms. Compatible with for depth-powered application development. Microsoft Kinect V1 OpenNI (No longer available new) — The Carmine series is near-identical in specification to the original Microsoft Kinect, and was available in short- and mid-range models. Unlike the original Microsoft Kinect, the Carmine does not include a microphone array. Compatible with for depth-powered application development. PrimeSense Carmine OpenNI — Ruggedized 3D sensors for industrial applications. The Photoneo phoXi line is available in multiple baselines. Photoneo — A range of structured light 3D sensors available in multiple baselines, with USB3 connection for the Zivid One series, and CAT 6A Ethernet connection for the Zivid Two. Zivid 3. Time of Flight Sensors Time of flight (ToF) sensors are effectively LiDAR sensors — aka, light radar. They send out packets of infrared light, and record the time it takes for the infrared light sent out to return. Despite using infrared light sources like structured light sensors, the manner in which ToF sensors transmit and receive light make them less susceptible to interference from bright and indirect sunlight. PMD Time of Flight (ToF) 3D sensor. ToFExpert, , via Wikimedia Commons CC BY-SA 3.0 ToF sensors often have wider range capabilities than structured light sensors, and can operate accurately for longer distance use cases. However, they aren’t able to return 3D images with the fidelity offered by structured light or stereo 3D sensors. Therefore, they are better suited for tasks like navigation assistance versus surface inspections. Examples: — The newest member of the Microsoft Kinect family of 3D sensors is a compact ToF module that adds a 12MP RGB video camera that is registered to the depth stream for high quality color imaging and textures. The Azure Kinect includes a built-in IMU for kinetic data. Microsoft Azure Kinect — Simple, compact ToF 3D sensors that can be purchased as embeddable modules, or as enclosed standalone sensors. PMD Pico — Increasingly hard-to-find. However, like the original Microsoft Kinect, PrimeSense Carmine and original Occipital Structure Sensor 3D sensors, the Xtion 2 is compatible with for depth-powered application development. ASUS Xtion 2 OpenNI Comparing Stereo Sensors to Structured Light to ToF Perception Sensors So why choose one modality over another? Well, we provided hints in the above descriptions for each modality, but let’s get more concrete on why you might choose one over another. At the highest level, the primary factors that determine 3D sensing selection are: Sensor Performance Impact on Host Environmental Limitations Let’s explore each of these factors in detail, and then show how each sensing modality compares. Sensor Performance The main parameters we’ll be exploring here are range, resolution and reliability. Range is perhaps the most important parameter, because different robot tasks are defined by the range in which they occur. For example, mapping environments at high speed for autonomous vehicles is typically left to rotating LiDAR units, as they have ranges that can exceed 100 meters, which allows them to see distant obstacles well before the autonomous vehicle is upon those obstacles. Those same LiDAR units, however, are poorly suited for close visual inspections, as these tasks benefit from sensors with sub-meter ranges that can support enhanced detail capture and accuracy. For stereo and structured light 3D sensors, range is a function of the baseline between the optical elements on the sensor itself. The rule of thumb is that the wider the baseline, the longer the range. Similarly, a narrower baseline creates a shorter range. So why not just spec a 3D sensor with the widest baseline possible? Well, for one, form factor limitations can preclude this approach. But, more importantly, wider baseline sensors’ sensing ranges start further away from the sensor. That means that a wide baseline sensor might ignore objects that are less than two meters from the sensor. A shorter baseline sensor might not have as long of a range, but it might be able to start sensing within .25 meters of the sensor. Therefore, baseline and the resultant range becomes a tradeoff of how near you need to sense versus how distant you need to sense. You get one or the other, but you can’t get both. Most 3D sensors have ranges that start at around 0.5–1m from the sensor, and extend to 3–6m at best. Keep in mind that at the furthest reaches of a sensor’s range, resolution and accuracy can start to diminish rapidly. ToF 3D sensors typically have much greater range than structured light or stereo 3D sensors, and can often receive reliable signals from objects that are up to 25m away. However, ToF sensors often suffer from excessive noise, and resolution can be much lower than similarly priced 3D modalities. Speaking of resolution, this is both a function of range, as well as the nature of the optics used by the sensor. Structured light sensors’ resolution is limited by the density of the pattern that they project (which gets less dense as distance increases), and the resolution of the IR camera that reads the pattern as it falls on objects and environments to be sensed. Stereo sensors’ resolution is limited by the native resolution of the cameras used for the sensor. For tasks such as navigation, high resolution isn’t strictly necessary, unless objects that might cause an obstruction are very fine (an extension cable, for instance, may be on the edge of detectability for some low-resolution 3D sensors). For tasks such as surface inspections and 3D modeling, high resolution is much more desirable. Because these tasks often occur at close range, many structured light and stereo sensors are well suited to this kind of task. ToF sensors are best suited for mid-distance navigation tasks, or skeletal tracking. These tasks don’t require high-resolution sensing, but benefit from the range and operational speed of ToF sensors. Lastly, let’s talk about reliability. For sensors, this comes in two forms: data acquisition reliability, and mechanical reliability. Data acquisition reliability comes down to the inherent limitations of each sensor modality. Stereo sensors, for instance, need sufficient ambient light in the visible spectrum to work properly. Conversely, infrared-based sensors like structured light and ToF suffer when exposed to too much ambient light, causing sensor washout and data integrity issues. These sensors also struggle with black surfaces (they absorb IR wavelengths) and shiny surfaces (they scatter projected IR light). Active stereo sensors with infrared pattern projectors offer a “best of both worlds” approach that broaden sensor reliability over a number of environments and lighting conditions. Mechanical reliability is often the result of compromises inherent to a specific modality, or result from manufacturer cost cutting. Many consumer-grade sensors use USB for data and power transfer; this is fine in a lab or test environment. However, in real-world use, USB can be flaky and error-prone, resulting in intermittent sensor failures and compromised data. Ethernet is much more stable, but is rarely found on consumer-grade sensors. It is much more likely to be found on much more expensive industrial-grade sensors, where it is a prerequisite for a purchase decision. Impact on Host The main parameters we’ll be exploring here are power consumption, compute use, heat dissipation, data transmission and form factor. All robotic hosts have finite amounts of compute and power available. Robots, drones and other vision-enabled devices are often resource balancing acts where multiple sensors, motors, actuators, and other components fight for scarce system resources. Different sensors have different impacts. Sensors with active IR components are much more power hungry than passive stereo 3d sensors, for instance. But passive stereo sensors with high dynamic range (HDR) cameras may use much more compute than a structured light sensor, as the images they capture are much more data rich. Thus, sensor decisions must be made holistically in context of the other components that require system resources (and that subject is complex enough to deserve its own blog post). Similarly, data transmission and storage can often be limited on robots and drones. Again, the example of using an HDR-equipped 3D sensor can strain even robust WiFi networks given the sheer size of the datastreams being captured and transmitted. Lastly, there are industrial design considerations that can limit what sensor options can be specified. These considerations include the size and weight of a sensor (particularly important for lightweight devices like drones), as well as a sensor’s ability to dissipate heat generated. The latter is of particular importance to IR-equipped sensors. The laser-based IR components in these sensors rely on operating within a specific temperature range to maintain the bandwidth at which they operate. Deviating by even a few degrees Fahrenheit outside of the operating temperature range can cause significant variations in sensor readings, corrupting the data generated enough to make it useless. Therefore, considerations must be made to ventilate or insulate these sensors, depending on the environments in which they will be used. Environmental Limitations The main parameters we’ll be exploring here are ambient lighting, and connectivity. We mentioned above that many of these sensors are sensitive to ambient lighting, but it bears repeating. Lighting challenges are one of the most common causes of 3D sensor failures for robots. A robot that worked exceptionally well in a laboratory environment with consistent lighting conditions will suddenly experience drastic and intermittent failures when placed in a real world context with changing, inconsistent lighting conditions. Therefore, it is important to know all the lighting permutations that may exist upon deployment, and account for those during sensor selection. Connectivity is also important. If sensor data is to be processed and used locally, then connectivity isn’t an issue. However, for robots that will be deployed in an environment with low bandwidth, then it is either necessary to compress or filter sensor data on the host before transmitting, or it is necessary to choose a low bandwidth sensor type (for instance, ToF). Again, it is important to test for packet size thresholds under real world bandwidth constraints, as laboratory environments may not sufficiently reveal deficiencies. Synthesizing the Data and Deciding We’ve put together a handy chart showing different , and the 3D sensors that work best for those scenarios. We’ve noted what key decision factors should be considered for each application, and added notes to further clarify why specific choices should be made. robot tasks to be done We hope that you’ve found this guide to 3D sensors useful! Our next sensor roundup will cover LiDAR sensors, so stay tuned. If you’re currently working on a vision-enabled product like a robot or a drone, please take a look at what we do at . We can help you get your product to market faster, and keep it functioning more reliably upon deployment, too. Tangram Vision Also published at https://medium.com/tangram-visions/sensors-101-3d-sensing-66e528f276c4

3 Common Types of 3D Sensors: Stereo, Structured Light, and ToF

This is a tutorial about how we can implement IBM Cloud Storage in our Ruby on Rails 6 projects. We can upload images and manage these assets with Rails. In this tutorial we are going to build a simple app. We can upload a kitty photo and vote for that photo (this tutorial only includes the part of creating a new Rails app, configuring it to save our photo on the cloud, and showing it on an image tag; the design and the part for the photo will be included in the app but is not shown in this tutorial). You can find the app on this repository , and the live version here: . https://github.com/redacuve/my-cat-photos https://my-cat-photos.herokuapp.com/ Before we start, here is a little bit of background. I needed to use file storage for my project on Rails 6. At first Amazon S3 seemed fair to me. There is a lot of documentation about how to use AWS, but there was a problem. For a free account you need a valid credit card. I was looking for other options but they all needed credit cards (Azure, Google Cloud). Then I found IBM Cloud. For a free account you don’t need a credit card. IBM also upgraded their cloud. Now it is compatible with protocol S3 (Simple Storage Service) like Amazon Web Services. Thanks to this protocol you can save objects (they can be files, or in this case images) and every object is saved in a bucket (a container of objects). This is great because the previous cloud of IBM was tricky to use. Creating an account and making our Bucket The first part is, of course, going to the registration URL and registering. We can go to and create our account. https://cloud.ibm.com/registration We need to provide our email and password. Then click on Next. We need to look for our confirmation email. IBM will send us a code and we need to insert that code in the given field and click on Next. We need to add our personal information and click on . Create the account When we create our account, we can read some documentation and we can click on . Then we are going to see the dashboard. Accept I’m not going to explain all of the services that IBM includes, but feel free to check them out. There are some very good services. We can create our resource clicking on “Create resource”. We are going to search “Object Storage” and click on it. There we can see the features of our plan (lite and free). We can add the name of the resource (I will call it my-cat-photos; you can call it whatever you want). And then select “Create”. Now we are on the Cloud Object Storage section. We need to select “Create Bucket”. Here we can customize our bucket clicking on Custom Bucket. But a standard predefined bucket is fine, so we can select that one. it’s important to have a unique bucket name. We can see the location of the bucket (and we can change it) but the default location is OK. We can also see the service credentials, where we can add another one later. We are ready to click on “View Bucket Configuration”. IMPORTANT. Here there is information that we will need later. We need to look at it and save it for later. Specifically, we need: The region Bucket name Endpoint. We can find the first two on bucket details and the last one scrolling a little bit to endpoints and copy the public one. For me, these are the data: us-south Region: my-cat-photos-cos-standard-cpm Bucket name: s3.us-south.cloud-object-storage.appdomain.cloud Endpoint: Now we can obtain the credentials. For that we need to click on the “Service credentials” menu on the left. Here we are going to click on the “New Credential” button. We can name the credential as we want. But we need to click on advanced options. Here we need to toggle on the switch to include the HMAC credential to our credential and click on add. Then our credential will be created. We can expand the credential and copy the values: and access_key_id secret_access_key At this point, we need to have the following: Region Bucket Name Endpoint access_key_id secret_access_key. With this information we can go on to the second part of this tutorial. Creating our app with Rails 6 This is simple. We are going to create the app. It’s straightforward. We need a model for the photo, a controller with 3 methods (index, new, and create), and two views. One for creating the photo and another for displaying all photos (index and new), and of course our model. In this model we need a field of the string type, because here is where we are going to save our image name. We also need to update the routes. So, let’s start. We can open a terminal in Linux (we need rails 6, bundler, and yarn). And hit this command to create our app: $ rails new my-cat-photos Inside the new folder that Rails creates for us, we can create the photos controller. $ rails generate controller photos Then, we can create the model. $ rails generate model Photo title:text description:text votes: image:string float We can start with the model ‘app/models/photo.rb’ working only with the validation of the image. In this case we need to have the title, the description, and the image to be present so we can save the record. validates_presence_of , , < ApplicationRecord class Photo :title :description :image end Then move to  the controller and the methods ‘app/controllers/photos_controller.rb’, it’s simple: @photos = Photo.all @photo = Photo.new @photo = Photo.new(photo_params) @photo.save
      flash[ ] = redirect_to root_path render private params. ( ).permit( , , , , ) < ApplicationController class PhotosController def index end def new end def create if :success 'Photo Created successfully' else :new end end def photo_params require :photo :title :description :votes :image :image_cache end end With the controller done, we can update our routes ‘config/routes.rb’. It only has three routes. The index route is the main route of our project. Rails.application.routes.draw root get , post , do # For details on the DSL available within this file, see https://guides.rubyonrails.org/routing.html 'photos#index' '/upload' to: 'photos#new' '/photos' to: 'photos#create' end Now we can work with the View. First the new view ‘app/views/photos/new.html.erb’ to create the photo. It's important to create the hidden field called _cache at the end. This is so if validation fails, we don't need to search the photo again. That's why this field is included in the “photo_params” function of the controller. We also need to put in the form multipart: true as an option of the HTML because we need to have the file uploaded in our form. <%= %> <% %> <%= %> <% %> <%= %> <% %> <% %> <%= %> <%= %> <%= %> <%= %> <%= %> <%= %> <%= %> <%= %> <%= %> <%= %> <% %> form_with( @photo, , { }) model: local: true html: multipart: true do |form| @photo.errors.any? if < > div < > h2 pluralize(@photo.errors.count, ) "error" </ > h2 < > ul @photo.errors.full_messages.each do |message| < > li message </ > li end </ > ul </ > div end < = > div class "field" form.label :title form.text_field :title </ > div < = > div class "field" form.label :description form.text_field :description </ > div < = > div class "field" form.label :votes form.number_field :votes </ > div < = > div class "field" form.label , :image 'Upload the foto' form.file_field :image </ > div form.hidden_field :image_cache < = > div class "actions" form.submit "Upload Your photo" </ > div end And the index view ‘app/views/photos/index.html.erb’ is simple. It only shows all the fields for every photo. At the end we have our link to the upload route (where there’s the form to upload our image). <% %> <%= %> <%= %> <%= %> <%= %> <% %> <%= %> @photos.each do |photo| < > ul < > li photo.title </ > li < > li photo.description </ > li < > li photo.votes </ > li < > li photo.image </ > li </ > ul end link_to , upload_path "Create new" Here is a simple app (don’t forget to run the migrations). We can create a photo and see all of the photos created, but if we tried to see something similar to an image it’s not going to work. We cannot see any photo, photo name, or URL. Instead in the photo.image field, we can see something similar to this We are going to solve this with the IBM Cloud. For this, we need to use three gems: #<ActionDispatch::Http::UploadedFile:0x00007fb0651ff3c0> Is a gem that simplifies for us the upload of files in ruby applications. We can see the readme here: . Carrierwave: https://github.com/carrierwaveuploader/carrierwave This gem will help us upload the file to the cloud. It uses the S3 protocol. IBM used the same protocol, so we are going to be able to upload and retrieve the images. The readme is here . Fog-aws: https://github.com/fog/fog-aws With this gem we can resize images before uploading, and we are going to use this to upload a thumb version of our main image. Here is the repository . MiniMagick: https://github.com/minimagick/minimagick We can put these gems in our gemfile. gem , gem gem # My Gems # Uploader 'carrierwave' '~> 2.0' # Help with Ibm Cloud (aws configuration) 'fog-aws' # image resizing 'mini_magick' We proceed to install these gems with this command: $ bundle install Configuring Carrierwave and Fow-aws This is the best part of this tutorial. For the configuration of carrierwave, fog-aws, and minimagick, to work with IBM Cloud, we need to have this: Region. Bucket Name. Endpoint. access_key_id. secret_access_key. The first step to do is generate the uploader for the image, the name of the uploader is up to you but I’m going to call it photo. $ rails generate uploader photo This command will generate a file called photo_uploader.rb inside the folder ‘app/uploaders’, this is a configuration file. Here we are going to change some things. Comment the storage :file to storage :fog (to use with our fow-aws gem). Uncomment the extension_whitelist function to only allow uploading images. Uncomment the version :thumb. This will use mini_magick to create a thumb version of our file. To use this gem we need to uncomment this line: ‘include CarrierWave::MiniMagick’. So our file ‘app/uploaders/photo_uploader.rb’ needs to look like this: CarrierWave::MiniMagick
  storage version process [ , ] < CarrierWave::Uploader:: class PhotoUploader Base include :fog def store_dir "uploads/ / / " #{model. .to_s.underscore} class #{mounted_as} #{model.id} end :thumb do resize_to_fit: 50 50 end def extension_whitelist %w(jpg jpeg gif png) end end After we configure our uploader we need to add it to the model. This is simple. We need to mount the uploader to one field on our model. That field needs to be a field of the string type (that's why when we generate our model we made a field called image of the string type) and attach the uploader; in this case, it’s the PhotoUploader. mount_uploader , PhotoUploader
  validates_presence_of , , < ApplicationRecord class Photo :image :title :description :image end Our next step is to configure fog (that line that we uncomment storage :fog). For that we need to create an initializer for fog. This file needs to be called carrierwave.rb and needs to be inside the folder ‘config/initializers’. So we are going to paste this placeholder code in our file ‘config/initializers/carrierwave.rb’. CarrierWave.configure config.fog_credentials = { , , , , , , }
  config.fog_directory  = config.fog_public     = config.fog_attributes = { } do |config| provider: 'AWS' # required aws_access_key_id: 'xxx' # required unless using use_iam_profile aws_secret_access_key: 'yyy' # required unless using use_iam_profile use_iam_profile: true # optional, defaults to false region: 'eu-west-1' # optional, defaults to 'us-east-1' host: 's3.example.com' # optional, defaults to nil endpoint: 'https://s3.example.com:8080' # optional, defaults to nil 'name_of_bucket' # required false # optional, defaults to true cache_control: "public, max-age= " #{ .days.to_i} 365 # optional, defaults to {} end Before configuring this code, we need to store our credentials in a safe place (if we upload the code with the credentials it can be very dangerous because in a public repo everyone can steal the credentials and that is a bad thing). So we are going to use Credentials. Credentials on Rails 6 Using credentials with Rails 6 is so simple. We have a file called credentials.yml.enc inside the folder ‘config’. This file is encrypted with our ‘master.key’ located in the same folder. That's why our master.key file always needs to be ignored for git. One use of the master.key (between many others) is to decrypt our credentials.yml.enc and use the information that is inside (generally used to store credentials of APIs). To access, decrypt, and edit credentials.yml.enc first we need our favorite editor (for example: gedit, emacs, geany, kate, kwrite, vim, mousepad, etc.) and use the name of the editor to edit the information. I’m going to use nano because I love it. We need to run this command in our terminal: $ EDITOR=’nano’ rails credentials:edit Now our favorite editor will display the information of credentials.yml.enc. Here we can simply uncomment these lines: # aws: # access_key_id: 123 # secret_access_key: 345 And add our keys without quotes, so its something like this aws:
  access_key_id: e5e4f3363d2dbd1aebf13630a26078d5
  secret_access_key: b083b1387037b87454298fc1067404b71feec88b5d945d25 We can save our file, and it will be encrypted. Now we can access that data in our Rails app. We just need to call: NameOfTheApp::Application [:keyToFetch] .credentials .nameOfTheCredential If we forget the name of the application we can find it here: ‘config/application.rb’. If I want to access key id for the name of the module (in my case is MyCatPhotos) I need this lines. MyCatPhotos::Application [:access_key_id]

MyCatPhotos::Application [:secret_access_key] .credentials .aws .credentials .aws Finish configuration Now it’s time to configure the initiliazer ‘config/initializers/carrierwave.rb’. We need to put the aws_secret_key_id and aws_secret_access_key from our encrypted credentials. use_iam_profile needs to be false (it’s the default). region is the region of our bucket. host will be nil (it’s the default). endpoint will be our endpoints. These are the parts of fog credentials. The last thing is the config.fog_directory. Here we need to put our bucket name and leave everything with its defaults. So our code needs to look like this: CarrierWave.configure config.fog_credentials = { , MyCatPhotos::Application.credentials.aws[ ], MyCatPhotos::Application.credentials.aws[ ], , , , }
    config.fog_directory  = config.fog_public     = config.fog_attributes = { } do |config| provider: 'AWS' aws_access_key_id: :access_key_id aws_secret_access_key: :secret_access_key use_iam_profile: false region: 'us-south' host: nil endpoint: 'https://s3.us-south.cloud-object-storage.appdomain.cloud' 'my-cat-photos-cos-standard-cpm' false cache_control: "public, max-age= " #{ .days.to_i} 365 end and we are almost done!. Every time we upload an image it’s going to be saved in our cloud. So to retrieve the image and show it on our page we need to access the property URL, and this will be the source of an image tag. In Rails, we can achieve that with <%= image_tag photo.image.url %> So we modify our index view a little bit, ‘app/views/photos/index.html.erb’, to show the image, if it exists. <% %> <%= %> <%= %> <%= %> <%= %> <% %> @photos.each do |photo| < > ul < > li photo.title </ > li < > li photo.description </ > li < > li photo.votes </ > li image_tag photo.image.url !photo.image.url. ? if nil </ > ul end Now our project can show the images that we upload to the cloud. We can also inspect the url of our images and they are created by default with carrierwave and fow. Looks something like this: https://my-cat-photos-cos-standard-cpm.s3.us-south.cloud-object-storage.appdomain.cloud/uploads/photo/image/1/bebudito.jpg?X-Amz-Expires=600&X-Amz-Date=20200623T205500Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=dbd1aebf108e4f33b3630a2e5d5363d2%2F20200623%2Fus-south%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=723299b8ee0e077a7c70265673825873002ed71838ea2ed28e8a9dc2066ad49f If we can retrieve the thumb version it’s with photo.image.thumb.url. This is the end of this tutorial. We left out a little bit of design and functionalities (like the voting). If you like I can make another tutorial about design and functionalities. I didn’t do it here because this tutorial is too long, but you can see the full app on my . repo

How To Upload and Display Images Trough IBM Cloud with Rails 6

It’s surprisingly difficult to know whether an email you’ve received is trustworthy. Why is this so hard? Well, you can’t trust the sender name. You can’t trust the email’s aesthetics. And you can’t trust the copy. To be safe, you need to verify the sender’s domain name and the domain of all outgoing links. definitely Put yourself in the shoes of a random user and try to decipher “efax.hosting.com.mailru.co” or “drive.google.com.download-photo.net”. It’s unreasonable to expect the average user to be on high alert every time they open their inbox, so let’s find a better way. Using color The fundamental problem is that the information you need to determine whether an email is safe is 1) hard to find and 2) hard to understand. In a perfect world, this information would be 1) easy to skim and 2) impossible to misread. There are a handful of ways to verify you’re talking with the right person. You can verify 1) something they are (biometrics), 2) something they have (2FA codes), or 3) something they know (passwords). It’s not immediately clear how we might associate emails with senders’ biometrics or 2FA codes, but we can definitely work off the idea of an “email password”. We provide websites with passwords to verify who we are all the time. What if websites needed to provide a password so that we could verify who they were? And what if these passwords weren’t complicated strings but rather easy-to-skim colors? us https://github.com/turbomaze/colorful-phish Imagine this: the moment you created a password for a website, they created one to use with you. In your welcome email, they told you what this password was: a specific color, unique to your account. From that moment forward, you could rest easy knowing that if an email didn’t contain that exact color, then you were getting phished. Here’s an example of what one of these welcome emails might look like: And we’re done. Colors are simple to implement, easy to skim, and impossible to misunderstand. Check out for a Node.js implementation that will help you eliminate phishing on your site with 3 lines of code. colorful-phish on GitHub If you have any thoughts or want to riff on related ideas, you can find me on Twitter at . https://twitter.com/@imigliu

How Color Can Prevent Your Users from Getting Phished

Enhancing Cross-Chain Communication The goal of Photon is to solve the issues that are caused by interoperability by enabling decentralized apps (dApps) to communicate with one another in a seamless way across many blockchains. This is true independent of whether or not the blockchains in question are based on Ethereum Virtual Machine (EVM) architectures. This objective is accomplished by the company via the provision of high-speed communications services that connect more than 15 blockchains. Furthermore, the company has aspirations to grow this network even further in the year to come. Introduction to Photon's Cross-Chain Messaging A new breakthrough is likely to make it simpler for decentralized applications (dApps) to communicate with one another across various blockchain networks. This is due to the fact that blockchain technology is always experiencing advancements. Recent experiments have shown that the Photon cross-chain message that Entangle is developing, which will be made available along with the mainnet of Entangle, is the most efficient signaling protocol in the Web3 environment. The company's innovative approach to cross-chain communications was what made this significant improvement in communication possible. It is a new feature known as "transmitter Agents" that is at the heart of the Photon protocol's ability to minimize the amount of time it takes for blocks to be created and to make the transaction process move more smoothly. The Need for Speed and Efficiency There are a plethora of independent networks that make up the world of blockchain technology. Each of these networks operates under its own set of restrictions. The concept that the value of a smart contract is directly proportional to the degree of its communication capabilities is the philosophical foundation upon which Photon's design philosophy is founded. It is very necessary to have effective communication throughout the various chains in order to make full advantage of the potential offered by Web3 technologies. Enhanced user experiences and more decentralized application capabilities will be made possible as a result of this occurrence. Technical Insights The Transmitter Agent framework is responsible for the technological foundations of Photon's high-speed messaging. This framework makes use of a distributed network of agents to identify and propose cross-chain communications in a proactive manner. Through collaborative efforts, these agents are able to reach an agreement on the authenticity of messages, which enables them to swiftly finish transactions that include many chains. In order to maintain a flexible infrastructure that can accommodate the ever-changing nature of blockchain activities, Photon has implemented lightweight off-chain components and decreased the amount of time it takes for blocks to be generated. Furthermore, developers are granted the liberty to design their own unique consensus thresholds, which further optimizes the validation process in order to enable complicated processes and provide a wide range of functionality. Photon's strategy for ensuring fast and reliable transfers is predicated on the optimized design of the Transmitter Agents, which are strategically distributed to facilitate rapid event capture and proposal submission. This design choice allows for quicker consensus among Transmitter Agents and timely execution of cross-chain messages. Overview of Entangle With the goal of creating a data infrastructure that is entirely adaptive and interoperable, Entangle is a larger project that wants to attract both Web3 applications and existing organizations. The instant supply of photon is not the only thing that this effort does. Entangle's goal is to provide blockchain-based applications and ecosystems communication channels for smart contracts that are not only cost-effective but also secure, interoperable, and allow quick transmission. This will be accomplished via the use of Photon as the company's flagship protocol. There are a wide range of real-world data sources that might potentially be affected by such a data solution. These include, but are not limited to, financial markets, digital asset exchanges, sports, gaming, and a great lot of other examples. Through the provision of developers with the capacity to access and interact with a broad variety of permitted data, Entangle is marketing itself as a supplier of complete data solutions in the Web3 area. Entangle is making this effort as part of its overall strategy to strengthen its position in the industry. Conclusion Photon offers fast speeds, freedom, and security, which will help Web3 dApps reach more people and work better. Photon wants to speed up the creation of a blockchain network environment that is more connected and flexible. Entangle wants to make cross-chain DeFi easier and better for all users. This shows that it is committed to making decentralized technology better and easier to use. Adding these kinds of unique rules will change blockchain and the digital market as it grows. Don’t forget to like and share the story! Vested Interest Disclosure: This author is an independent contributor publishing via our brand-as-author program. Be it through direct compensation, media partnerships, or networking, the author has a vested interest in the company/ies mentioned in this story. HackerNoon has reviewed the report for quality, but the claims herein belong to the author. #DYOR

Cross-Chain Communication Hits Lightspeed with Entangle's Photon

Machine learning is expensive - it requires high computing capacity. For example, training of the OpenAI’s famous GPT-3 model , and requires an amount of energy equivalent to the and creates a carbon footprint equivalent to traveling 700,000 kilometers by car for a single training session. would cost over $4.5M yearly consumption of 126 Danish homes Photonic processors are one of the most promising solutions to the huge computing capacity need and the extreme carbon footprint because they are fast (as we know, light is the fastest thing in the universe) and really energy efficient. Some companies on the market have production-ready photonic processors. One of my favorites is LightOn because their solution is relatively easy, and the way it works is something like magic (you will see why). can do only one thing, but it can do this one thing fast and efficiently. It can multiply a vector with a giant random matrix. The hardware converts the input to a light pattern. A special waveguide does the random matrix multiplication in an analog way. In the end, a camera converts back the result to a vector. The random matrix is fixed, so the multiplication hardware (the waveguide) here is a passive element. But how can help a random matrix multiplicator to machine learning? LightOn’s OPU (Optical Processing Unit) Multiplication with a fixed giant random matrix sounds meaningless, but don’t listen to your intuition, it is a very meaningful thing. It is a convenient way for dimension reduction called random projection. The random projection has a good property; it nearly preserves the distances because of the . So if you have two vectors in a high dimensional space, the distance of the projected vectors will be nearly the same as their original distance. Johnson–Lindenstrauss lemma This distance preserving property can be used very well in reinforcement learning. In LightOn’s example, they are using their OPU to convert the actual game state to a 32 dimension vector. Here the game state is a screen snapshot that is represented by a 33600 (210x160 pixels) vector—the OPU projects from this 33600 dimension space to a 32 dimension space. The learning method is simple Q-learning. The algorithm calculates the Q value and stores it for the (projected) state. If the Q value is unknown, it uses the Q value of the 9 nearest neighbors and calculates the average of them. Because of the Johnson–Lindenstrauss lemma, the distance of the projected vectors is more or less the same as the original state vectors, so the K nearest neighbors of the projected vector are projected from near original vectors. In this case, OPU is used to reduce the dimensions to create an easier problem (a 32 dimension vector is better than a 33600 dimension) which needs less storage, less computing capacity, or a smaller neural network. The same method can be used in recommender systems or any other cases where dimension reduction can be used, and the OPU does it fast on really high dimensional vectors. You can find the code . here Another use case, when OPU is used to project from a low dimensional space to a high dimensional space. In LightOn’s transfer learning example, OPU is used instead of the linear layers of a neural network. In this case, a pretrained CNN (VGG, ResNet, etc.) is used for feature extraction, but the classification is not done by a neural network. Instead of the dense layers, OPU is projecting the feature vector to a high dimensional space where the samples are linearly separable. So, thanks to OPU a simple linear classifier can be used instead of a neural network. The above 2 examples used the OPU to map the original problem to an easier problem that needs less computing capacity and storage, but can it be used to make the neural network training process easier? The answer is yes. There is a method called in this case the . This method has 2 advantages. We can train the hidden layers parallelly and the error is propagated through a fixed random matrix what can be done efficiently with the OPU. Using random matrices instead of backpropagation sounds like magic, but it works. Direct Feedback Alignment error is propagated through fixed random feedback connections directly from the output layer to each hidden layer Here is a nice YouTube video about . DFA As you saw, a relatively simple photonic hardware can help multiple ways to reduce the huge computing capacity needs of machine learning, and OPU is only one of the photonic hardware on the market. There are other solutions with programable matrices or quantum photonic solutions. It is worth keeping your eyes on this field because machine learning needs more and more computing capacity, and photonic processors are one of the most promising ways to provide this.

How Photonic Processors Will Save Machine Learning

You can apply any design, lighting, or graphics style to your 4K image in real-time using this new machine learning-based approach! If you think this looks interesting, watch the video on this topic and read more about it from the references below 👇 Watch the video References ►Read the full article: https://www.louisbouchard.ai/4k-image-translation-in-real-time/ ►Liang, Jie and Zeng, Hui and Zhang, Lei, (2021), "High-Resolution Photorealistic Image Translation in Real-Time: A Laplacian Pyramid Translation Network", https://export.arxiv.org/pdf/2105.09188.pdf ►Code: https://github.com/csjliang/LPTN Video Transcript 00:00 You've all seen these kinds of pictures where a person's face is "toonified" into an anime 00:05 character. 00:06 Many of you must have seen other kinds of image transformations like this, where an 00:10 image is changed to follow the style of a certain artist. 00:13 Here, an even more challenging task could be something like this, where an image is 00:18 transformed into another season or time of the day. 00:21 What you have not seen yet is the time it takes to produce these results and the actual 00:25 resolutions of the produced pictures. 00:28 This new paper is completely transparent towards this as it attacks exactly this problem. 00:33 Indeed, compared to most approaches, they translate high-definition 4K images, and this 00:38 is done in real-time. 00:39 In this work, they showed their results on season translation, night and day translations, 00:45 and photo retouching, which you've been looking at for the last minute. 00:49 This task is also known as 'image-to-image translation', and all the results you see 00:53 here were produced in 4K. 00:55 Of course, this video is not in 4K, and the images were taken from their paper, so it 01:00 might not look that high-quality here. 01:03 Please look at their paper or try their code if you are not convinced! 01:07 These are the most amazing results of this paper. 01:09 Here, you can see their technique below called LPTN, which stands for Laplacian Pyramid Translation 01:16 Network. 01:17 Look at how much less time it took LPTN to produce the image translations where most 01:22 approaches cannot even do it as this amount of definition is just too computationally 01:27 demanding. 01:28 And yes, this is in seconds. 01:30 They could translate 4K images in not even a tenth of a second using a single regular 01:36 GPU. 01:37 It is faster than all these approaches on 480p image translations! 01:41 And yes, it is not eight times faster, but 80 times faster on average! 01:46 But how is that possible? 01:48 How can they be so much more efficient and still produce amazing and high-quality results? 01:53 This is achieved by optimizing the fact that illumination and color manipulation, which 01:58 relates to the style of an image, is contained in the low-frequency component of an image. 02:03 Whereas the content details, which we want to keep when translating an image into another 02:08 style, can be adaptively refined on high-frequency components. 02:13 This is where it becomes interesting. 02:15 These two components can be divided into two tasks that can be performed simultaneously 02:19 by the GPU. 02:21 Indeed, they split the image into low-resolution and high-resolution components, use a network 02:26 to process the information of the low-frequency or the style of the image, 02:30 and render a final image merging this processed style with the refined high-frequency component, 02:37 which is the details of the image but adapted by a smaller sub-network to fit the new style. 02:43 Thus dodging the unavoidable heavy computation consumption when processing the high-resolution 02:48 components in the whole network. 02:50 This has been a long-standing studied field achieved with a popular technique called Laplacian 02:55 Pyramid. 02:57 The main idea of this Laplacian Pyramid method is to decompose the image into high and low-frequency 03:02 segments and reconstruct it afterward. 03:05 First, we produce an average of the initial image, making it blurry and removing high-frequency 03:11 components. 03:12 This is done using a kernel that passes through the whole image to round batches of pixels 03:17 together. 03:18 For example, if they take a 3 by 3 kernel, it would go through the whole image averaging 03:23 3 by 3 patches removing all unique values. 03:26 They are basically blurring the image by softening the edges. 03:30 Then, the difference between this blurry image and the initial image is saved to use at the 03:35 end of the algorithm to re-introduce the details, which are the high-frequency components. 03:41 This is repeated three times with bigger and bigger averaging kernels producing smaller 03:47 and smaller low-frequency versions of the image having less and less high-frequency 03:52 details. 03:53 If you remember, these low-frequency versions of the image contain information about the 03:57 colors in the image and illumination. 03:59 Indeed, they are basically just a blurred low-quality version of our image, which is 04:03 why the model is so much more efficient. 04:06 This is convenient since they are smaller versions of the image, and this is the exact 04:11 information we are trying to change when translating the image into another style. 04:16 Meaning that using these low-frequency versions is much more computationally efficient than 04:21 using the whole image directly, but they are also focused on the information we want to 04:26 change in the image, which is why the results are so great. 04:30 This lower-quality version of the image can be easily translated using an encoder-decoder, 04:35 just like any other image translation technique we previously mentioned, but since it is done 04:40 on a much lower quality image and a much smaller image, 04:43 it is exponentially faster to process. 04:47 The best thing is that the quality of the results only depends on the initially saved 04:52 high-frequency versions of the image sent as input which is not processed throughout 04:57 the whole network. 04:59 This high-frequency information is simply merged at the end of the processing with the 05:03 low-frequency image to improve the details. 05:06 Basically, it is so much faster because the researchers split the image's information 05:11 in two: low-frequency general information and detailed high-frequency information. 05:17 Then, they send only the computational-friendly part of the image, which is exactly what we 05:23 want to transform, the blurry, low-quality general style of the image, or in other words: 05:29 the low-frequency information. 05:30 Then, only fast and straightforward transformations are done on the high-frequency parts of the 05:36 image to resize them and merge them with the blurry newly-stylized image, 05:42 improving the results by adding details on all edges in the picture. 05:46 And voilà! 05:47 You have your results with a fraction of the time and computational power needed. 05:51 This is brilliant, and the code is publicly available if you would like to try it, which 05:56 is always cool! 05:58 As always, the links to the complete article and references are in the description of the 06:02 video. 06:03 Thank you for watching!

High-Resolution Photorealistic Image Translation in Real Time

Is AI the future of video game design? This video is about a paper called Enhancing Photorealism Enhancement. Enhancing Photorealism Enhancement, or EPE, is an AI that can be applied live to the video game and transform every frame to look much more natural. You can find out more about it below. Watch the video References: ►Read the full article: https://www.louisbouchard.ai/the-future-of-video-game ►Richter, Abu AlHaija, Koltun, (2021), "Enhancing Photorealism Enhancement", https://intel-isl.github.io/PhotorealismEnhancement/ Video Transcript 00:00 What you see here is the gameplay of a very popular video game called GTA 5. 00:04 It looks super realistic, but it is still obvious that this is a video game. 00:09 Now, look at this... 00:10 No, this is not real life. 00:13 It is still the same GTA5 gameplay that went through a new model using artificial intelligence 00:18 to enhance its graphics and make it look more like the real world! 00:23 The researchers from Intel Labs just published this paper called Enhancing Photorealism Enhancement. 00:29 And if you think that this may be "just another GAN," taking a picture of the video game as 00:35 an input and changing it following the style of the natural world, let me change your mind. 00:40 They worked on this model for two years to make it extremely robust. 00:44 It can be applied live to the video game and transform every frame to look much more natural. 00:50 Just imagine the possibilities where you can put a lot less effort into the game graphic, 00:54 make it super stable and complete, then improve the style using this model. 00:59 I think this is a massive breakthrough for video games, and this is just the first paper 01:03 attacking this same task applied specifically to video games! 01:07 I want to ask you a question that you can already answer or wait until the end of the 01:11 video to answer: Do you think this is the future of video games? 01:16 If you want more time to answer, that's perfect, let's get into this technique. 01:21 In general, this task is called image-to-image translation. 01:24 You take an image and transform it into another, often using GANs, as I covered numerous times 01:30 in my previous videos. 01:31 If you want an overview of how a typical GAN architecture works, I invite you to check 01:35 out this video appearing on the top right corner as I won't get into the details of 01:40 how it works here. 01:42 As I said earlier, this model is different than basic image-to-image translation as it 01:47 uses the fact that it is applied to a video game. 01:50 This is of enormous importance here as video games have much more information than a simple 01:55 picture, so why make the task more complicated by achieving realistic transformations using 02:01 only the snapshot as input? 02:03 Instead, they use much more information already available for each image of the game like 02:08 the surface normals, depth information, materials, transparency, lighting, and even a segmentation 02:15 map which tells you what and where the objects are. 02:18 I'm sure you can already see how all this additional information can help with this 02:22 task. 02:23 All these images are sent to a first network called the G-buffer Encoder. 02:29 This G-buffer encoder takes all this information, sends it into a classic convolutional network 02:34 independently to extract and condense all the valuable information from these different 02:40 versions of the initial image. 02:42 This is done using multiple residual blocks, as you can see here, which is basically just 02:47 a convolutional neural network architecture, and more precisely, a ResNet architecture. 02:52 The information is extracted at multiple steps, as you can see. 02:56 This is done to obtain information at different stages of the process. 03:00 Early information is vital in this task because it gives more information regarding the spatial 03:05 location and has the smaller details information. 03:08 In comparison, deeper information is essential to understand the overall image and its style. 03:14 A combination of both early and deep information is thus very powerful when used correctly! 03:20 Then, all this information here, referred to as the G-buffer features, is sent to another 03:25 network with the original image from the game called the rendered image. 03:30 You can see the different colors representing the G-buffer information extracted from different 03:35 scales as we previously saw, with the gray arrow showing the process for the actual image. 03:40 Here again, you can see this as an enhanced version of the same residual blocks as for 03:46 the g-buffer encoder repeated multiple times, but with a little tweak to better adapt the 03:51 G-buffer information before being added to the process. 03:54 This is done using what they refer to as RAD here, which is again residual blocks, convolutions, 04:00 and normalization. 04:02 As I mentioned, this architecture is a bit more complicated than a simple encoder-decoder 04:06 architecture like regular GAN. 04:08 Similarly, the training process is also more elaborated. 04:12 Here, you can see two metrics, the realism score, and the LPIPS score. 04:17 The realism score is basically the GAN section of the training process. 04:22 It compares both a similar real-world image to a game image and compares the real image 04:28 to an enhanced game image. 04:29 Helping the model to learn how to produce a realistic and enhanced version of the game 04:34 image sent. 04:36 Whereas this LPIPS component is a known loss used to retain the structure of the rendered 04:41 image as much as possible. 04:43 This is achieved by giving a score based on the difference between the associated pixels 04:48 of the rendered image versus the enhanced image. 04:51 Penalizing the network when it generates a new image that spatially differs from the 04:56 original image. 04:58 So both these metrics work together to improve the overall results during the training of 05:03 this algorithm. 05:04 Of course, as always, you need a large enough dataset of the real world and of the game 05:10 as it won't generate something that the model has never seen before. 05:14 And now, do you think this kind of model is the future of video games? 05:18 Has your opinion changed after seeing this video? 05:21 As always, the references are linked in the description below, and the full article is 05:25 available on my website louisbouchard.ai with more information. 05:29 Thank you for watching!

This AI by Intel Makes Grand Theft Auto 5 Look Photorealistic

DeOldify is a technique to colorize and restore old black and white images or even film footage. It was developed and is still getting updated by only one person, Jason Antic. It is now the state-of-the-art way to colorize black and white images. Best of all, everything is open-sourced! Watch the video References What are GANs: ​ https://youtu.be/ZnpZsiy_p2M GitHub with full code, in-depth explanation, and Colabs: ​ https://github.com/jantic/DeOldify Free DeepAI image colorization API using DeOldify: ​ https://deepai.org/machine-learning-m... MyHeritage colorization tool (the best version of DeOldify available, paid): ​ https://www.myheritage.com/incolor Follow me for more AI content: Instagram: ​ https://www.instagram.com/whats_ai/ LinkedIn: ​ https://www.linkedin.com/in/whats-ai/ Twitter: ​ https://twitter.com/Whats_AI Facebook: ​ https://www.facebook.com/whats.artifi... The best courses in AI: ​ https://www.omologapps.com/whats-ai Join Our Discord channel, Learn AI Together: ​ https://discord.gg/learnaitogether Chapters 0:00​ - Hey! Tap the Thumbs Up button and Subscribe. You'll learn a lot of cool stuff, I promise. 0:40​ - DeOldify explanation 3:22​ - How to use DeOldify Yourself 4:24​ - Old Movie Examples 5:06​ - Conclusion Video transcript 00:00 this ai can colorize and restore old 00:03 black and white images and even film 00:05 footage 00:06 this method is called de-aldefy and 00:09 works on pretty much 00:10 any picture if you don't believe me you 00:13 can even try it yourself for free as i 00:15 will show in the video 00:17 but first let's see how it works and 00:19 more amazing results to convince you to 00:22 try it 00:25 [Music] 00:29 this is what's ai and i share artificial 00:31 intelligence news every week 00:33 if you are new to the channel and want 00:34 to stay up to date please consider 00:36 subscribing to not miss any further news 00:40 the aldephi is a technique to colorize 00:42 and restore 00:43 old black and white images or even film 00:45 footage 00:46 it was developed and is still getting 00:48 updated by only one person 00:50 jason antique it is now the 00:53 state-of-the-art way to colorize black 00:55 and white images 00:56 and everything is open sourced but we 00:59 will get back to this in a bit 01:01 first let's see how he achieved that it 01:04 uses a new type of gun training method 01:07 called nogan that he developed himself 01:09 to solve the main problems that appeared 01:11 when training using a normal 01:13 adversarial network architecture 01:15 composed of a discriminator and a 01:17 generator 01:18 typically gun training works by both 01:21 training the discriminator and generator 01:23 at the same time 01:24 where the generator starts by being 01:26 completely random 01:27 and improves over time to fool the 01:29 discriminator 01:30 which tries to tell if the image is 01:32 generated or real 01:34 if this was just completely abstract to 01:36 you i invite you to watch the video i 01:38 made about cans 01:39 in the upper right corner right now and 01:41 linked in the description 01:43 before continuing this video his new 01:46 method 01:46 which he calls the nogan provides the 01:49 same benefits of this usual gun training 01:52 while having to spend way less time 01:54 training the gan architecture 01:56 which is typically pretty heavy in 01:58 computation time 01:59 instead he pre-trains the generator to 02:02 make it already more powerful 02:04 fast and reliable using a regular loss 02:07 function 02:08 this is done by training the generator 02:10 like a regular deep networks 02:12 architecture 02:13 such as resnet that way the model is 02:16 already pretty good at colorizing an 02:18 image 02:18 before training the complete gan 02:20 architecture 02:22 then it only needs a short amount of 02:24 this typical generator discriminator 02:26 gan training to optimize the realism of 02:29 the generated pictures 02:32 gaussian noise is also randomly applied 02:34 to images to generate 02:36 fake noise during training this is a 02:39 type of data augmentation that can be 02:41 performed on the training images to 02:43 improve the results and resistance 02:45 to noisy inputs using the same technique 02:48 as style transfer where the noise will 02:50 be the style of the image we want to 02:52 copy 02:53 and can be applied more or less to the 02:55 transformation 02:57 the whole architecture uses a basic 02:59 resnet backbone 03:00 on a u-net where the generator network 03:02 in the gun training 03:04 is the unet architecture right now 03:07 there is no complete explanation of how 03:09 this works 03:10 but the author is currently working on a 03:13 paper about 03:14 the aldephi where he will further 03:16 investigate 03:17 why and how his technique previously 03:20 found only by trials and errors 03:22 work you can find three things in the 03:25 description of the video 03:27 at first there's the github link with a 03:30 complete detailed explanation of the 03:31 technique and even google collab 03:33 tutorials 03:34 to use it yourself just look at how 03:37 simple this is 03:38 you can just run the few sections enter 03:41 the link of your image 03:43 and run it 04:04 then you can find a free api on deep ai 04:08 using dl defy where you can simply click 04:11 and try yourself 04:13 finally the third link is the most 04:15 advanced version of the aldify if you 04:17 are looking for the best results 04:20 it is on myheritage's website and is 04:23 paid to use 04:24 let's just take a minute to see how it 04:26 works on old movies before ending this 04:40 video 04:50 you better get on the job some of the 04:52 kids may be up this afternoon 04:53 oh jack we can get along without 04:55 dragging those young kids up here oh why 04:57 don't you button up your lip 04:59 you're always squawking about something 05:00 you got more static on the radio 05:07 please leave a like if you went this far 05:09 in the video 05:10 and since there are over 90 of you guys 05:13 watching that are not subscribed yet 05:15 consider subscribing to the channel to 05:17 not miss any further news clearly 05:19 explained 05:20 thank you for watching 05:23 [Music]

DeOldify can Colorize your Black & White Photos with Full Photorealistic Renders

Water your plants, get alarms, and analyze the data Having plants at home is a luxury that only households that  always count with someone's presence can enjoy. If you live alone and travel for a few days, you are likely to return home and find a mumified version of your beloved basil plant. One may argue: "Why not buy a timer on Home Depot for 5 bucks?". That's an excellent point, but why not make it overly complicated and learn a couple of things during the process? In this article I will show you how I created an IoT application that controls the irrigation of the plants in my house. This application independently irrigates 5 pots: 1 lemon tree, 2 passion fruit plants, 1 gardenia and 1 pot with herbs (basil, fennel and oregano). Because I'm planting these plants on the balcony of my apartment, I don't have running water, so I'm using a 40 liters tank. This tank has an ultrasonic sensor that monitors the volume and once the water level is below 30%, my Losant app sends me an email telling me to fill the tank. Well, to be honest, it sends an email to my wife, I hate filling that tank. Besides that, the system gets the current temperature and humidity here in Houston, TX and based on that increases the volume of irrigation water to compensate for the extra heat. I realized that I needed to implement this feature once I was done with the hardware, so I'm pulling the temp info from a free weather API. It's pretty cool that Losant allows you to integrate info from so many different sources. The hardware The general system diagram is shown below: The electronic enclosure The electronics are very simple, these are the components of this project: Microprocessor: Particle Photon (https://www.particle.io) Power supplies (5 and 12 Volts) 6 Relays. Caveat: I have used mechanical relays because I've also used a switching power supply. SSD relays can switch as fast as your switching power supply and mess up with the output. Wifi antenna The basic idea is the following: 5V feed the Particle Photon Photon's outputs connect to the low power side of the relays driving 12V to the valve solenoids upon activation. Similarly, the pump (underwater aquarium pump) is controlled by the Photon which drives AC current through its motor. Lets put everything together in a nice, presentable box. Remember, if you use a metallic enclosure like I did, you need to ground the box. I can't stress this enough! The Water Manifold I got the cheapest solenoid valves I could get from AliExpress.com, I think I paid 3 bucks for each. This is not bad at all, in the US they go for 10 bucks each. I also got some valves, tubing and an aquarium manifold. I've put everything together in a fancy plastic box. Note that the electrical connection from this wet module to the electronics in done by a multi-core cable and a connector. The final result is pretty neat. Firmware The Particle Photon, like Arduino, uses C++. It has a some conveniences like Particle.function() and Particle.publish(). These features allow the user to trigger functions from external events and publish local events respectively. In order to trigger functions remotely you'll need to declare them in the void setup. In my system I have 7 Particle functions, they are: tooHot: changes the value of the boolean variable hot to TRUE if the local temperature is greater than 30 Celsius. The data is retrieved from the weather API and the logic is executed in Losant. mode: activates the manual mode. pf1, lemon, herbs, gardenia, pf2: water each plant in manual mode. These Particle functions only accept one argument and it must be a String. Conversaly, you can publish events with Particle.publish(). Please check out my code below and next we will see how to create an interface to interact with the microcontroller. V1 = ; V2 = ; V3 = ; V4 = ; V5 = ; pump = ; trigger = ; echo = ; volume; hot = FALSE; manual_mode = FALSE; { Particle.function( , tooHot);
    Particle.function( , mode);
    Particle.function( , pf1);
    Particle.function( , lemon);
    Particle.function( , herbs);
    Particle.function( , gardenia);
    Particle.function( , pf2); pinMode(trigger, OUTPUT);
    pinMode(echo, INPUT);
    pinMode(pump, OUTPUT);
    pinMode(V1, OUTPUT);
    pinMode(V2, OUTPUT);
    pinMode(V3, OUTPUT);
    pinMode(V4, OUTPUT);
    pinMode(V5, OUTPUT); Time.zone( ); digitalWrite(V1, HIGH);
    digitalWrite(V2, HIGH);
    digitalWrite(V3, HIGH);
    digitalWrite(V4, HIGH);
    digitalWrite(V5, HIGH);
    digitalWrite(pump, HIGH);
    
    Serial.begin( );
} { STARTUP(WiFi.selectAntenna(ANT_EXTERNAL)); (manual_mode == FALSE){ (Time.hour() == && Time.minute() == ){
            auto_irrigation();
        } (hot == TRUE){ (Time.hour() == && Time.minute() == ){
                auto_irrigation();
            }
        }
    } ((Time.hour() == || Time.hour() == ) && Time.minute() == ){
        volume = measure_level();
        Particle.publish( , (String)volume);
    }

} {
    digitalWrite(V1, LOW);
    digitalWrite(pump, LOW);
    delay( );
                
    digitalWrite(V1, HIGH);
    digitalWrite(V2, LOW);
    delay( );
                
    digitalWrite(V2, HIGH);
    digitalWrite(V3, LOW);
    delay( );
                
    digitalWrite(V3, HIGH);
    digitalWrite(V4, LOW);
    delay( );
                
    digitalWrite(V4, HIGH);
    digitalWrite(V5, LOW);
    delay( );
                
    digitalWrite(V5, HIGH);
    digitalWrite(pump, HIGH);
} { result; duration = ; ( i = ; i <= ; i++)
    {
        digitalWrite(trigger, LOW);
        delayMicroseconds( );
        digitalWrite(trigger, HIGH);
        delayMicroseconds( );
        digitalWrite(trigger, LOW);
        duration = duration + (pulseIn(echo, HIGH)/ );
        delay( );
    }
    result = *( -duration* )/ ; result;
} { (command == ){
        hot = TRUE; ;
    } (command == ){
        hot = FALSE; ;
    } { ;
    }
    
} { (command == ){
        manual_mode = TRUE; ;
    } (command == ){
        manual_mode = FALSE; ;
    } { ;
    }
} { (manual_mode == TRUE){ (command == ){
            digitalWrite(V1, LOW);
            digitalWrite(pump, LOW);
        } (command == ){
            digitalWrite(V1, HIGH);
            digitalWrite(pump, HIGH);
        }
    }
} { (manual_mode == TRUE){ (command == ){
            digitalWrite(V2, LOW);
            digitalWrite(pump, LOW);
        } (command == ){
            digitalWrite(V2, HIGH);
            digitalWrite(pump, HIGH);
        }
    }
} { (manual_mode == TRUE){ (command == ){
            digitalWrite(V3, LOW);
            digitalWrite(pump, LOW);
        } (command == ){
            digitalWrite(V3, HIGH);
            digitalWrite(pump, HIGH);
        }
    }
} { (manual_mode == TRUE){ (command == ){
            digitalWrite(V4, LOW);
            digitalWrite(pump, LOW);
        } (command == ){
            digitalWrite(V4, HIGH);
            digitalWrite(pump, HIGH);
        }
    }
} { (manual_mode == TRUE){ (command == ){
            digitalWrite(V5, LOW);
            digitalWrite(pump, LOW);
        } (command == ){
            digitalWrite(V5, HIGH);
            digitalWrite(pump, HIGH);
        }
    }
} // ports to control and read hardware const int 0 const int 1 const int 2 const int 3 const int 4 const int 5 const int 6 const int 7 // tank volume measurement int // temperature alarm bool // manual mode bool void setup () // declaring particle functions "tooHot" "mode" "pf1" "lemon" "herbs" "gardenia" "pf2" // defining i/o's // setting time zone -5 // initiallizing actuators 9600 void loop () // selects the u.FL antenna if // morning irrigation schedule if 7 00 // if it's warm, the system water the plants again at 3pm if if 15 00 // measure water tank level twice a day if 8 16 00 "tank_volume" /* FUNCTIONS */ // irrigation int auto_irrigation () 80000 50000 80000 40000 40000 // measure the tank level, averages 100 measurements int measure_level () int long 0 for int 0 100 2 10 100 50 100 25.4 0.017 25.4 return /* this function changes the boolean value of the variable hot and its logic is executed on Losant. 
the weather api provides the temp and if greater than 30C, it will trigger toohot. */ int tooHot (String command) if "yes" return 1 else if "no" return 0 else return -1 // this function changes the logic value of manual_mode and it's triggered on losant. int mode (String command) if "true" return 1 else if "false" return 0 else return -1 // passion fruit 1 manual irrigation int pf1 (String command) if if "true" else if "false" // lemon tree manual irrigation int lemon (String command) if if "true" else if "false" // herbs manual irrigation int herbs (String command) if if "true" else if "false" // gardenia manual irrigation int gardenia (String command) if if "true" else if "false" // passion fruit 2 manual irrigation int pf2 (String command) if if "true" else if "false" IoT Workflow Receive an email notification Losant is an IoT platform that allows the developper to create a workflow logic that will receive and trigger events. It's beyond the scope of this article to provide a detailed Losant tutorial, however I'll explain the intuition behind it and give you references in case you decide to develop your own IoT app. Let's start checking how to implement an email notification to alert the user that the water level is getting low. First you'll need to create a device and a particle integration in Losant. Upon doing that, go to your workflow and build the following block logic. Let's examinate how to set the nodes to implement this logic. This node establishes communication with your device. Click on it and select the integration you have created. 1. Particle: Remember Particle.publish()? We will set up this node to listen for events published from the Photon. In order to do that, we need to define the structure of the payload with some javascript. The event published by the microcontroller will be transmitted in a json package, more precisely inside data. In our case, the variable being passed is 'volume', so we can retrieve this data by accessing 'data.volume'. 2. Function: Click on the function node and add the following code. parts = payload.data.data.split( );
payload.data.volume = parts[ ]; var ':' 0 : This node will attribude the data from you payload to a device and the attributed you've defined upon its creation. This will allow us to create a dashboard later. 3. Device Get Select the query method to: 'Match all tags query' Set 'Key Templates' to 'particle_device_id' and 'Value Templates' to '{{ data.coreid }}' Set 'Result Path' to 'device_to_update' : Stores the state in a device attribute. 4. Device State 'Device ID JSON path' = 'device_to_update.id' 'Attribute' = '{{data.volume}}' This block allows you to debug and understand what is going on with you payload. You can trigger events from the workflow and see how your payload is being constructed. This block is not necessary to make the app work. 5. Debug: Executes a conditional logic. Remember you'll need to refer to the variable according to the way that your JSON payload was defined, in our case {{data.volume}}. 6. Conditional: This one is pretty self explanatory. Upon the activation of the condition previously defined, this block will send an email. 7. Email: To summarize, we will retrieve the variable 'volume' by setting up a logic to listen to the events being published from the IoT microcontroller. Upon getting this data from a JSON package we will compare it to a threshold and if the value is less than 30% of the total volume, an email will be sent asking the user to re-fill the tank. Get local temperature from an API First you'll need an API key, go to and create an account. You can use their free plan for this project. If I'm not wrong, the free plan includes 1000 api calls a day. If you're not in the agricultural business, this should be more than enough. https://openweathermap.org/api Build the following workflow: The http block is responsible from calling the API, let's check on how to set it up. In the 'URL template', add the following: http://api.openweathermap.org/data/2.5/weather?q='YOUR CITY'&appid='YOUR API KEY' The 'request method' should be set to GET and the API response should be stored in the 'Payload path to store response' field. Set it to 'working.weather'. Set the timer to call the API every hour so you don't exceed the daily limit of calls. Now your payload should look like this: Now the temperature should be accessible through: {{working.weather.body.main.temp}} Triggering Functions Remotely Now it's time to finally trigger our Particle functions previously defined in our firmware. Let's start by building a workflow: Basically we have a 'Virtual Button' node triggering a 'Particle Call' node for each one of our remotely operated functions. In the 'Particle Call' node, we need to: Inform the node which firmware function we will call by adding its name to the field 'Function Name'. Tell the node what argument we will be passing by specifying it directly or adding a payload path. We will pass a payload path since we will be using a toggle switch to activate the system. We will create variables for each plant and add them to the payload. They will be located at the root data, so they will be accessible as data.pf1, data.lemon, data.herbs, data.gardenia and data.pf2. Now, we need to create an interface to trigger this workflow we've just created. Go to Dashboard and add an input control. Add toggles and buttons like so: The toggle state is sent via payload to the microcontroller upon triggering the workflow via the button. Set the button like so (note, the value between the double brackets should be the same of your toggle, my toggle is also named 'lemon'). Do the same for all the other inputs and test it: Data visualization Add some charts and link then to your devices attributes. Conclusion I hope you guys enjoyed this article. This project has several moving parts and I know that I have covered some of them too fast without providing the level of detail that the reader deserves, for that reason do not hesitate to contact me if you'd like to get more info or simply ask a question. Moving forward I'd like to add a strain gauge to monitor the weight of the plant. Over 98% of the mass of plants comes from the air (interesting fact isn't it?). That would be an interesting step into starting optimizing fertilization and irrigation. If you are interested in learning more about Losant, check this tutorial out: https://www.losant.com/blog/how-to-integrate-particle-with-losant See y'all soon!

IoT Home Irrigation System Using Losant and Particle Photon

A while ago I was a mobile developer. I made apps for iOS and had to constantly worry about memory consumption, performance, you know, all the boring stuff that makes computers hard. Then I became a web developer. I kicked up my feet and let the VM do the work for me. Memory? Psh, buy more ram cheapskate. leaks? Who keeps a web page open long enough to have leaks anyway? Stop streaming videos from your work laptop and use a Chromecast, that’s an at-home-so-no-one-knows-you-love-the-bachelor type activity anyway. Memory I also have two dogs. They’re awesome, but I always felt like they didn’t have enough cool shit to call their own. Sure they have toys that they enjoy eating and later pooping out, but I’m a tech guy, and they’re tech-less dogs, which is a little embarrassing. So, naturally, I started trying to make a badass dog collar for the little luddite fluff balls. I picked up a few Particle Photons (which are awesome, highly recommend,) and was off to the races, in baby C and C++ just like data structures in college, except without the hangover, and the video games. Okay, still with the hangover, just without the video games. Okay, still with the video games. programming Programming in those old languages wasn’t so bad, they have pretty normal features, Strings are a pain, but the included libraries provide some good support there. Then, when I thought everything was going well, Bachelor in Paradise went and got shut down — oh, also my program kept crashing mysteriously. Turns out the issue was a memory leak in the code because, of course, I haven’t concerned myself with manual memory management in . years I continued on, trying to remember to alloc and dealloc like a champion, until my good friend, Will, suggested I try out Rust. He described it as “The right way to do memory management,” and I can now say I agree. That’s when I started trying to port my dog collar code to Rust. A big, huge, enormous shoutout to , who helped me a ton through all of this. The guy has the patience of Chris Harrison during a rose ceremony. Jorge Aparicio I started at , and was able to get my code up and running quickly. I was honestly surprised at how easy it was. I also needed to use a couple of libraries that were written in C/C++ to get the basic functionality going. That’s when I stumbled and fell down the rabbit hole. Jorge’s awesome quickstart readme I had some good guidance from Jorge on how to do this, basically there were a few steps to take: Get the C/C++ library Add the and crates to your project. bindgen gcc Make a build.rs file Add your build.rs file under in the cargo.toml file like so: [package] build = "build.rs" Using , generate the rust bindings from the header file of the library you want to import. In other words, add something like this to your build file’s main function: bindgen let bindings = bindgen::Builder::default().no_unstable_rust().header("src-cpp/some-library.hpp").generate().expect("Unable to generate bindings"); let out_path = PathBuf::from("src");bindings.write_to_file(out_path.join("lib.rs")).expect("Couldn't write bindings!"); My code just happens to reside in . src-cpp Finally, add the step that actually compiles the C/C++ code into a lib and puts it in your build target: gcc::Config::new().cpp(true) // Switch to C++ library compilation..file("src-cpp/add.cpp").compile("libsome-library.a"); If you’re lucky, that’s it! The library you’re including doesn’t have any references to the particle firmware and you’re all hunky dory. You can use it like a normal module now or require it in as an external crate, go crazy. If you didn’t get that ring, don’t worry, you can still go on Bachelor in Paradise to find true love, it’s just a little harder and you have to get really drunk in Mexico — I mean, add a bunch of compiler flags. Rust on the photon is great, but, like Bachelor in Paradise, it lacks standard(s)… libraries (got ‘em.) That’s fine, until you realize that practically relies on the standard library. That means that you have to be very careful after running your compile commands to audit the outputted rust code. everything I tried the process out with the and this is what my build.rs looked like after some tinkering: Adafruit Neopixel Library You’ll notice a few things here: There are a metric shit-ton of flags. Those are all the flags I threw in so that gcc would use the correct libraries and compile based on the Photon firmware build process. I basically ran the build process and copied the gcc step manually. I commented out the bindgen steps. This is because I used them to generate the rust bindings with basic types (being new to rust, this seemed like the safest thing,) and then I removed anything referencing the standard library or that seemed extraneous. There were, for example, a lot of things generated that were used by the Photon library, but not the bindings. This all seems well and good, and I believe that it was a great start, but, alas, like the proposals in the bachelor, the code crashed as soon as I tried it out for realsies. I’m still not sure what the issue was, perhaps it was that the Neopixel library uses raw assembly and Rust didn’t like that. Regardless, I ended up writing t, but perhaps I found a new love in embedded Rust. I accept this rose — I mean Rust. most of the functionality from scratch in Rus

Rust for the Particle Photon

Singularity: Streamlining Game Development with a Universal Backend Framework

এককতা: একটি ইউনিভার্সাল ব্যাকএন্ড ফ্রেমওয়ার্ক সহ গেম ডেভেলপমেন্টকে স্ট্রীমলাইন করা

Singularity: Optimierte Spieleentwicklung mit einem universellen Backend-Framework

Singularidad: agilización del desarrollo de juegos con un marco de backend universal

Singularity : rationaliser le développement de jeux avec un framework backend universel

सिंगलेरिटी: यूनिवर्सल बैकएंड फ्रेमवर्क के साथ गेम डेवलपमेंट को सुव्यवस्थित करना

Singularity: ユニバーサル バックエンド フレームワークによるゲーム開発の効率化

Singularity: 범용 백엔드 프레임워크로 게임 개발 간소화

Singularidade: simplificando o desenvolvimento de jogos com uma estrutura de back-end universal

Singularity: оптимизация разработки игр с помощью универсальной серверной платформы

Tekillik: Evrensel Arka Uç Çerçevesiyle Oyun Geliştirmeyi Kolaylaştırma

Điểm kỳ dị: Hợp lý hóa quá trình phát triển trò chơi với Khung phụ trợ phổ quát

Singularity：使用通用后端框架简化游戏开发

Authors:
(1) Han Jiang, HKUST and Equal contribution (hjiangav@connect.ust.hk);
(2) Haosen Sun, HKUST and Equal contribution (hsunas@connect.ust.hk);
(3) Ruoxuan Li, HKUST and Equal contribution (rliba@connect.ust.hk);
(4) Chi-Keung Tang, HKUST (cktang@cs.ust.hk);
(5) Yu-Wing Tai, Dartmouth College, (yu-wing.tai@dartmouth.edu). Table of Links Abstract and 1. Introduction 2. Related Work 2.1. NeRF Editing and 2.2. Inpainting Techniques 2.3. Text-Guided Visual Content Generation 3. Method 3.1. Training View Pre-processing 3.2. Progressive Training 3.3. 4D Extension 4. Experiments and 4.1. Experimental Setups 4.2. Ablation and comparison 5. Conclusion and 6. References ABSTRACT Current Neural Radiance Fields (NeRF) can generate photorealistic novel views. For editing 3D scenes represented by NeRF, with the advent of generative models, this paper proposes Inpaint4DNeRF to capitalize on state-of-the-art stable diffusion models (e.g., ControlNet [39]) for direct generation of the underlying completed background content, regardless of static or dynamic. The key advantages of this generative approach for NeRF inpainting are twofold. First, after rough mask propagation, to complete or fill in previously occluded content, we can individually generate a small subset of completed images with plausible content, called seed images, from which simple 3D geometry proxies can be derived. Second and the remaining problem is thus 3D multiview consistency among all completed images, now guided by the seed images and their 3D proxies. Without other bells and whistles, our generative Inpaint4DNeRF baseline framework is general which can be readily extended to 4D dynamic NeRFs, where temporal consistency can be naturally handled in a similar way as our multiview consistency 1. INTRODUCTION Recent development of Neural Radiance Fields (NeRF) [18] and its dynamic variants including [20, 25, 28] have shown their great potential in modeling scenes in 3D and 4D. This representation is suitable for scene editing from straightforward user inputs such as text prompts, eliminating the need for modeling and animating in detail. One important task in scene editing is generative inpainting, which refers to generating plausible content that is consistent with the background scene. Generative inpainting has a wide range of potential applications, including digital art creation, and VR/AR. Although several recent works have addressed the inpainting and text-guided content generation problem on NeRFs, they have various limitations in directly generating consistent novel content seamlessly with the background based on the text input. Specifically, [4, 24, 41] allow users to edit the appearance of an existing object in the NeRF based on the text prompt, but their generations are limited by the original target object, thus they cannot handle substantial geometry changes. In [19, 31] a given NeRF is inpainted by removing the target object and inferring the background, but their inpainting task is not generative as they do not match the inferred background with any other user input. Recent generative works [12, 21, 26, 32] produce new 3D or 4D content from the text input, but the generation is not conditioned on the existing background. While [27] can generate contents around the target object without removing it, generative inpainting on NeRF should allow users to remove target objects while filling the exposed region with plausible 3D content, which was previously totally or partially occluded in the given scene. While applying recent diffusion models [22] on generative 2D inpainting may largely solve the problem, simply extending diffusion models to higher dimensions to static or dynamic NeRFs introduces extensive challenges, including new and more complex network structures and sufficient higher-dimensional training data. On the other hand, NeRFs are continuous multiview estimations on its training images which provide a natural connection between 2D images and NeRF, enabling us to propagate the 2D inpainting results by diffusion models to the underlying scene. Thus, in this paper, we propose Inpaint4DNeRF, the first work on text-guided generative NeRF inpainting with diffusion models, which can be naturally extended to inpaint 4D dynamic NeRFs. Given a text prompt, and a target foreground region specified by the user by text or on the single individual image(s), our method first utilizes stable diffusion to inpaint a few independent training views. Next, we regard these inpainted views as seed images and infer a coarse geometry proxy from them. With the strong guidance by seed images and their geometry proxies, the other views being inpainted are constrained to be consistent with the seed images when refined with stable diffusion, which is followed by NeRF finetuning with progressive updates on the training views to obtain final multiview convergence. For dynamic cases, once generative inpainting is achieved on a static single frame, it can be regarded as the seed frame where the edited information can be propagated to other frames to naturally inpaint the scene in 4D without bells and whistles. Our generative Inpaint4DNeRF proposes a text-guided approach for inpainting the relevant background after removing static foreground objects specified by a user-supplied prompt, where multiview consistency is meticulously maintained. To summarize, Inpaint4DNeRF presents the following contributions. First, harnessing the power of recent diffusion models on image inpainting, we can directly generate textguided content with new geometry while being consistent with the context given by the unmasked background. Second, our approach infers and refines other views from initially inpainted seed images to achieve multiview consistency across all the given views, where temporal consistency is naturally enforced and achieved. This paper is available on arxiv under CC 4.0 license. Authors: (1) Han Jiang, HKUST and Equal contribution (hjiangav@connect.ust.hk); (2) Haosen Sun, HKUST and Equal contribution (hsunas@connect.ust.hk); (3) Ruoxuan Li, HKUST and Equal contribution (rliba@connect.ust.hk); (4) Chi-Keung Tang, HKUST (cktang@cs.ust.hk); (5) Yu-Wing Tai, Dartmouth College, (yu-wing.tai@dartmouth.edu). Authors: Authors: (1) Han Jiang, HKUST and Equal contribution (hjiangav@connect.ust.hk); (2) Haosen Sun, HKUST and Equal contribution (hsunas@connect.ust.hk); (3) Ruoxuan Li, HKUST and Equal contribution (rliba@connect.ust.hk); (4) Chi-Keung Tang, HKUST (cktang@cs.ust.hk); (5) Yu-Wing Tai, Dartmouth College, (yu-wing.tai@dartmouth.edu). Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2. Related Work 2.1. NeRF Editing and 2.2. Inpainting Techniques 2.1. NeRF Editing and 2.2. Inpainting Techniques 2.3. Text-Guided Visual Content Generation 2.3. Text-Guided Visual Content Generation 3. Method 3. Method 3.1. Training View Pre-processing 3.1. Training View Pre-processing 3.2. Progressive Training 3.2. Progressive Training 3.3. 4D Extension 3.3. 4D Extension 4. Experiments and 4.1. Experimental Setups 4. Experiments and 4.1. Experimental Setups 4.2. Ablation and comparison 4.2. Ablation and comparison 5. Conclusion and 6. References 5. Conclusion and 6. References ABSTRACT Current Neural Radiance Fields (NeRF) can generate photorealistic novel views. For editing 3D scenes represented by NeRF, with the advent of generative models, this paper proposes Inpaint4DNeRF to capitalize on state-of-the-art stable diffusion models (e.g., ControlNet [39]) for direct generation of the underlying completed background content, regardless of static or dynamic. The key advantages of this generative approach for NeRF inpainting are twofold. First, after rough mask propagation, to complete or fill in previously occluded content, we can individually generate a small subset of completed images with plausible content, called seed images, from which simple 3D geometry proxies can be derived. Second and the remaining problem is thus 3D multiview consistency among all completed images, now guided by the seed images and their 3D proxies. Without other bells and whistles, our generative Inpaint4DNeRF baseline framework is general which can be readily extended to 4D dynamic NeRFs, where temporal consistency can be naturally handled in a similar way as our multiview consistency Current Neural Radiance Fields (NeRF) can generate photorealistic novel views. For editing 3D scenes represented by NeRF, with the advent of generative models, this paper proposes Inpaint4DNeRF to capitalize on state-of-the-art stable diffusion models (e.g., ControlNet [39]) for direct generation of the underlying completed background content, regardless of static or dynamic. The key advantages of this generative approach for NeRF inpainting are twofold. First, after rough mask propagation, to complete or fill in previously occluded content, we can individually generate a small subset of completed images with plausible content, called seed images, from which simple 3D geometry proxies can be derived. Second and the remaining problem is thus 3D multiview consistency among all completed images, now guided by the seed images and their 3D proxies. Without other bells and whistles, our generative Inpaint4DNeRF baseline framework is general which can be readily extended to 4D dynamic NeRFs, where temporal consistency can be naturally handled in a similar way as our multiview consistency direct 1. INTRODUCTION Recent development of Neural Radiance Fields (NeRF) [18] and its dynamic variants including [20, 25, 28] have shown their great potential in modeling scenes in 3D and 4D. This representation is suitable for scene editing from straightforward user inputs such as text prompts, eliminating the need for modeling and animating in detail. One important task in scene editing is generative inpainting, which refers to generating plausible content that is consistent with the background scene. Generative inpainting has a wide range of potential applications, including digital art creation, and VR/AR. Although several recent works have addressed the inpainting and text-guided content generation problem on NeRFs, they have various limitations in directly generating consistent novel content seamlessly with the background based on the text input. Specifically, [4, 24, 41] allow users to edit the appearance of an existing object in the NeRF based on the text prompt, but their generations are limited by the original target object, thus they cannot handle substantial geometry changes. In [19, 31] a given NeRF is inpainted by removing the target object and inferring the background, but their inpainting task is not generative as they do not match the inferred background with any other user input. Recent generative works [12, 21, 26, 32] produce new 3D or 4D content from the text input, but the generation is not conditioned on the existing background. While [27] can generate contents around the target object without removing it, generative inpainting on NeRF should allow users to remove target objects while filling the exposed region with plausible 3D content, which was previously totally or partially occluded in the given scene. While applying recent diffusion models [22] on generative 2D inpainting may largely solve the problem, simply extending diffusion models to higher dimensions to static or dynamic NeRFs introduces extensive challenges, including new and more complex network structures and sufficient higher-dimensional training data. On the other hand, NeRFs are continuous multiview estimations on its training images which provide a natural connection between 2D images and NeRF, enabling us to propagate the 2D inpainting results by diffusion models to the underlying scene. Thus, in this paper, we propose Inpaint4DNeRF, the first work on text-guided generative NeRF inpainting with diffusion models, which can be naturally extended to inpaint 4D dynamic NeRFs. Given a text prompt, and a target foreground region specified by the user by text or on the single individual image(s), our method first utilizes stable diffusion to inpaint a few independent training views. Next, we regard these inpainted views as seed images and infer a coarse geometry proxy from them. With the strong guidance by seed images and their geometry proxies, the other views being inpainted are constrained to be consistent with the seed images when refined with stable diffusion, which is followed by NeRF finetuning with progressive updates on the training views to obtain final multiview convergence. For dynamic cases, once generative inpainting is achieved on a static single frame, it can be regarded as the seed frame where the edited information can be propagated to other frames to naturally inpaint the scene in 4D without bells and whistles. Our generative Inpaint4DNeRF proposes a text-guided approach for inpainting the relevant background after removing static foreground objects specified by a user-supplied prompt, where multiview consistency is meticulously maintained. To summarize, Inpaint4DNeRF presents the following contributions. First, harnessing the power of recent diffusion models on image inpainting, we can directly generate textguided content with new geometry while being consistent with the context given by the unmasked background. Second, our approach infers and refines other views from initially inpainted seed images to achieve multiview consistency across all the given views, where temporal consistency is naturally enforced and achieved. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

NeRF Editing and Inpainting Techniques: Abstract and Introduction

Authors:
(1) Han Jiang, HKUST and Equal contribution (hjiangav@connect.ust.hk);
(2) Haosen Sun, HKUST and Equal contribution (hsunas@connect.ust.hk);
(3) Ruoxuan Li, HKUST and Equal contribution (rliba@connect.ust.hk);
(4) Chi-Keung Tang, HKUST (cktang@cs.ust.hk);
(5) Yu-Wing Tai, Dartmouth College, (yu-wing.tai@dartmouth.edu). Table of Links Abstract and 1. Introduction 2. Related Work 2.1. NeRF Editing and 2.2. Inpainting Techniques 2.3. Text-Guided Visual Content Generation 3. Method 3.1. Training View Pre-processing 3.2. Progressive Training 3.3. 4D Extension 4. Experiments and 4.1. Experimental Setups 4.2. Ablation and comparison 5. Conclusion and 6. References 3.2. Progressive Training Warmup Training. Our training image pre-processing stage provides a good initialization for rough convergence. Before fine-tuning on these images to get fine convergence, notice that the 3D object in NeRF is still the original object intact, which is quite different from what is depicted in the pre-processed training images above in geometry and appearance. Therefore, for the first stage of NeRF training, without any finetuning, we directly train the NeRF on the preprocessed images for coarse convergence. Note that when we perform fine-tuning later, the original object will have no effect because warmup training has already erased all its appearance information. Iterative Dataset Update. Given a NeRF in which the editing target has fixed overall geometry, Iterative Dataset Update (IDU), first proposed and shown effective in InstructNerf2Nerf [4], is a useful training image fine-tuning strategy which can edit the appearance and fine geometry of the target object in NeRF. In our task, warmup training has already provided a converged coarse geometry, leaving fine geometry and appearance to be determined. In their work, they use InstructPix2Pix [1] as a diffusion model strongly conditioned on the original training image. In our task, we use stable diffusion in a similar way to achieve similar objectives. The fine-tuned image should be conditioned on the pre-processed image, as well as the depth map. In detail, for pre-processed image conditioning, we adopt the same approach as in the previously introduced projected image correction. The preprocessed image and the current NeRF rendering are blended together, either in image space or in latent space. Then a small amount of noise is injected into the blended image, followed by stable diffusion denoising from an intermediate timestep. For depth map conditioning, we input the depth map of the current NeRF into ControlNet to guide the editing process. Along with training image updates and NeRF training, the underlying NeRF will gradually converge to an object with fine details, making rendering from NeRF more reliable. Meanwhile, stable diffusion correction with less noise injected into the training images introduces less multiview inconsistency to fine details. Therefore, we gradually increase the intermediate timestep of our stable diffusion correction when training NeRF, until the intermediate timestep becomes very close to the final denoise timestep. Consequently, the coarse NeRF converges to a fine one. Regularizers. To supervise NeRF training, we use the L1 photometric loss between NeRF rendering and the training images. However, sole supervision of RGB cannot avoid inaccuracies and noise in geometry. To achieve clean converged geometry, we add the following two regularizers into supervision. First, we also render a depth value along a ray, and compute the depth loss as defined in [3] between the rendered depth and our pre-processed 2-layer depth i.e., initial inpainted geometry proxies. Supervision with our planar 2-layer depth is not accurate, but it helps to avoid incorrect converged geometry, such as merging of the inpainted geometry with the background. Second, to deal with noisy RGB and depth renderings and “floaters” (artifacts hovering in the underlying NeRF volume), we use LPIPS loss [40] on RGB as a regularizing term. In practice, we found LPIPS to be effective in reducing noise and floaters. Since LPIPS is based on patches, rays selected in NeRF training are not allowed to be random, but are required to form rectangular patches. Thus we render small image patches, corresponding to rectangular areas in ground truth images, during NeRF optimization. The effect of the regularizers is also previously demonstrated in [31, 34]. This paper is available on arxiv under CC 4.0 license. Authors: (1) Han Jiang, HKUST and Equal contribution (hjiangav@connect.ust.hk); (2) Haosen Sun, HKUST and Equal contribution (hsunas@connect.ust.hk); (3) Ruoxuan Li, HKUST and Equal contribution (rliba@connect.ust.hk); (4) Chi-Keung Tang, HKUST (cktang@cs.ust.hk); (5) Yu-Wing Tai, Dartmouth College, (yu-wing.tai@dartmouth.edu). Authors: Authors: (1) Han Jiang, HKUST and Equal contribution (hjiangav@connect.ust.hk); (2) Haosen Sun, HKUST and Equal contribution (hsunas@connect.ust.hk); (3) Ruoxuan Li, HKUST and Equal contribution (rliba@connect.ust.hk); (4) Chi-Keung Tang, HKUST (cktang@cs.ust.hk); (5) Yu-Wing Tai, Dartmouth College, (yu-wing.tai@dartmouth.edu). Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2. Related Work 2.1. NeRF Editing and 2.2. Inpainting Techniques 2.1. NeRF Editing and 2.2. Inpainting Techniques 2.3. Text-Guided Visual Content Generation 2.3. Text-Guided Visual Content Generation 3. Method 3. Method 3.1. Training View Pre-processing 3.1. Training View Pre-processing 3.2. Progressive Training 3.2. Progressive Training 3.3. 4D Extension 3.3. 4D Extension 4. Experiments and 4.1. Experimental Setups 4. Experiments and 4.1. Experimental Setups 4.2. Ablation and comparison 4.2. Ablation and comparison 5. Conclusion and 6. References 5. Conclusion and 6. References 3.2. Progressive Training Warmup Training. Our training image pre-processing stage provides a good initialization for rough convergence. Before fine-tuning on these images to get fine convergence, notice that the 3D object in NeRF is still the original object intact, which is quite different from what is depicted in the pre-processed training images above in geometry and appearance. Therefore, for the first stage of NeRF training, without any finetuning, we directly train the NeRF on the preprocessed images for coarse convergence. Note that when we perform fine-tuning later, the original object will have no effect because warmup training has already erased all its appearance information. Warmup Training. Iterative Dataset Update. Given a NeRF in which the editing target has fixed overall geometry, Iterative Dataset Update (IDU), first proposed and shown effective in InstructNerf2Nerf [4], is a useful training image fine-tuning strategy which can edit the appearance and fine geometry of the target object in NeRF. In our task, warmup training has already provided a converged coarse geometry, leaving fine geometry and appearance to be determined. In their work, they use InstructPix2Pix [1] as a diffusion model strongly conditioned on the original training image. In our task, we use stable diffusion in a similar way to achieve similar objectives. The fine-tuned image should be conditioned on the pre-processed image, as well as the depth map. In detail, for pre-processed image conditioning, we adopt the same approach as in the previously introduced projected image correction. The preprocessed image and the current NeRF rendering are blended together, either in image space or in latent space. Then a small amount of noise is injected into the blended image, followed by stable diffusion denoising from an intermediate timestep. For depth map conditioning, we input the depth map of the current NeRF into ControlNet to guide the editing process. Iterative Dataset Update. Along with training image updates and NeRF training, the underlying NeRF will gradually converge to an object with fine details, making rendering from NeRF more reliable. Meanwhile, stable diffusion correction with less noise injected into the training images introduces less multiview inconsistency to fine details. Therefore, we gradually increase the intermediate timestep of our stable diffusion correction when training NeRF, until the intermediate timestep becomes very close to the final denoise timestep. Consequently, the coarse NeRF converges to a fine one. Regularizers. To supervise NeRF training, we use the L1 photometric loss between NeRF rendering and the training images. However, sole supervision of RGB cannot avoid inaccuracies and noise in geometry. To achieve clean converged geometry, we add the following two regularizers into supervision. First, we also render a depth value along a ray, and compute the depth loss as defined in [3] between the rendered depth and our pre-processed 2-layer depth i.e., initial inpainted geometry proxies. Supervision with our planar 2-layer depth is not accurate, but it helps to avoid incorrect converged geometry, such as merging of the inpainted geometry with the background. Second, to deal with noisy RGB and depth renderings and “floaters” (artifacts hovering in the underlying NeRF volume), we use LPIPS loss [40] on RGB as a regularizing term. In practice, we found LPIPS to be effective in reducing noise and floaters. Since LPIPS is based on patches, rays selected in NeRF training are not allowed to be random, but are required to form rectangular patches. Thus we render small image patches, corresponding to rectangular areas in ground truth images, during NeRF optimization. The effect of the regularizers is also previously demonstrated in [31, 34]. Regularizers. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

NeRF Editing and Inpainting Techniques: Progressive Training

The Chosen One: Consistent Characters in Text-to-Image Diffusion Models: References

Authors:
(1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem;
(2) Amir Hertz, Google Research;
(3) Yael Vinker, Google Research and Tel Aviv University;
(4) Moab Arar, Google Research and Tel Aviv University;
(5) Shlomi Fruchter, Google Research;
(6) Ohad Fried, Reichman University;
(7) Daniel Cohen-Or, Google Research and Tel Aviv University;
(8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Table of Links Abstract and Introduction
Related Work
Method
Experiments
Limitations and Conclusions
A. Additional Experiments
B. Implementation Details
C. Societal Impact
References 4. Experiments In Section 4.1 we compare our method against several baselines, both qualitatively and quantitatively. Next, in Section 4.2 we describe the user study we conducted and present its results. The results of an ablation study are reported in Section 4.3. Finally, in Section 4.4 we demonstrate several applications of our method. 4.1. Qualitative and Quantitative Comparison We compared our method against the most related personalization techniques [20, 42, 71, 89, 93]. In each experiment, each of these techniques is used to extract a character from a single image, generated by SDXL [57] from an input prompt p. The same prompt p is also provided as input to our method. Textual Inversion (TI) [20] optimizes a textual token using several images of the same concept, and we converted it to support SDXL by learning two text tokens, one for each of its text encoders, as we did in our method. In addition, we used LoRA DreamBooth [71] (LoRA DB), which we found less prone to overfitting than standard DB. Furthermore, we compared against all available image encoder techniques that encode a single image into the textual space of the diffusion model for later generation in novel contexts: BLIP-Diffusion [42], ELITE [89], and IP-adapter [93]. For all the baselines, we used the same prompt p to generate a single image, and used it to extract the identity via optimization (TI and LoRA DB) or encoding (ELITE, BLIP-diffusion and IP-adapter). In Figure 5 we qualitatively compare our method against the above baselines. While TI [20], BLIP-diffusion [42] and IP-adapter [93] are able to follow the specified prompt, they fail to produce a consistent character. LoRA DB [71] succeeds in consistent generation, but it does not always respond to the prompt. Furthermore, the resulting character is generated in the same fixed pose. ELITE [90] struggles with prompt following and the generated characters tend to be deformed. In comparison, our method is able to follow the prompt and maintain consistency, while generating appealing characters in different poses and viewing angles. In order to automatically evaluate our method and the baselines quantitatively, we instructed ChatGPT [53] to generate prompts for characters of different types (e.g., animals, creatures, objects, etc.) in different styles (e.g., stickers, animations, photorealistic images, etc.). Each of these prompts was then used to extract a consistent character by our method and by each of the baselines. Next, we generated these characters in a predefined collection of novel contexts. For a visual comparison, please refer to the supplementary material. We employ two standard evaluation metrics: prompt similarity and identity consistency, which are commonly used in the personalization literature [6, 20, 70]. Prompt similarity measures the correspondence between the generated images and the input text prompt. We use the standard CLIP [61] similarity, i.e., the normalized cosine similarity between the CLIP image embedding of the generated images and the CLIP text embedding of the source prompts. For measuring identity consistency, we calculate the similarity between the CLIP image embeddings of generated images of the same concept across different contexts. As can be seen in Figure 6 (left), there is an inherent trade-off between prompt similarity and identity consistency: LoRA DB and ELITE exhibit high identity consistency, while sacrificing prompt similarity. TI and BLIPdiffusion achieve high prompt similarity but low identity consistency. Our method and IP-adapter both lie on the Pareto front. However, our method achieves better identity consistency than IP-adapter, which is significant from the user’s perspective, as supported by our user study. 4.2. User Study We conducted a user study to evaluate our method, using the Amazon Mechanical Turk (AMT) platform [2]. We used the same generated prompts and samples that were used in Section 4.1 and asked the evaluators to rate the prompt similarity and identity consistency of each result on a Likert scale of 1–5. For ranking the prompt similarity, the evaluators were presented with the target text prompt and the result of our method and the baselines on the same page, and were asked to rate each of the images. For identity consistency, for each of the generated concepts, we compared our method and the baselines by randomly choosing pairs of generated images with different target prompts, and the evaluators were asked to rate on a scale of 1–5 whether the images contain the same main character. Again, all the pairs of the same character for the different baselines were shown on the same page. As can be seen in Figure 6 (right), our method again exhibits a good balance between identity consistency and prompt similarity, with a wider gap separating it from the baselines. For more details and statistical significance analbaselines. For more details and statistical significance analysis, read the supplementary material. 4.3. Ablation Study We conducted an ablation study for the following cases: (1) Without clustering — we omit the clustering step described in Section 3.1, and instead simply generate 5 images according to the input prompt. (2) Without LoRA — we reduce the optimizable representation Θ in the identity extraction stage, as described in Section 3.2, to consist of only the newly-added text tokens without the additional LoRA weights. (3) With re-initialization — instead of using the latest representation Θ in each of the optimization iterations, as described in Section 3.3, we re-initialize it in each iteration. (4) Single iteration — rather than iterating until convergence (Section 3.3), we stop after a single iteration. As can be seen in Figure 6 (left), all of the above key components are crucial for achieving a consistent identity in the final result: (1) removing the clustering harms the identity extraction stage because the training set is too diverse, (2) reducing the representation causes underfitting, as the model does not have enough parameters to properly capture the identity, (3) re-initializing the representation in each iteration, or (4) performing a single iteration, does not allow the model to converge into a single identity. For a visual comparison of the ablation study, as well as comparison of alternative feature extractors (DINOv1 [14] and CLIP [61]), please refer to the supplementary material. 4.4. Applications As demonstrated in Figure 7, our method can be used for various down-stream tasks, such as (a) Illustrating a story by breaking it into a different scenes and using the same consistent character for all of them. (b) Local text-driven image editing by integrating Blended Latent Diffusion [5, 7] — a consistent character can be injected into a specified location of a provided background image, in a novel pose specified by a text prompt. (c) Generating a consistent character with an additional pose control using ControlNet [97]. For more details, please refer to the supplementary material. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem; (2) Amir Hertz, Google Research; (3) Yael Vinker, Google Research and Tel Aviv University; (4) Moab Arar, Google Research and Tel Aviv University; (5) Shlomi Fruchter, Google Research; (6) Ohad Fried, Reichman University; (7) Daniel Cohen-Or, Google Research and Tel Aviv University; (8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Authors: Authors: (1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem; (2) Amir Hertz, Google Research; (3) Yael Vinker, Google Research and Tel Aviv University; (4) Moab Arar, Google Research and Tel Aviv University; (5) Shlomi Fruchter, Google Research; (6) Ohad Fried, Reichman University; (7) Daniel Cohen-Or, Google Research and Tel Aviv University; (8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Table of Links Abstract and Introduction Related Work Method Experiments Limitations and Conclusions A. Additional Experiments B. Implementation Details C. Societal Impact References Abstract and Introduction Abstract and Introduction Related Work Related Work Method Method Experiments Experiments Limitations and Conclusions Limitations and Conclusions A. Additional Experiments A. Additional Experiments B. Implementation Details B. Implementation Details C. Societal Impact C. Societal Impact References References 4. Experiments In Section 4.1 we compare our method against several baselines, both qualitatively and quantitatively. Next, in Section 4.2 we describe the user study we conducted and present its results. The results of an ablation study are reported in Section 4.3. Finally, in Section 4.4 we demonstrate several applications of our method. 4.1. Qualitative and Quantitative Comparison We compared our method against the most related personalization techniques [20, 42, 71, 89, 93]. In each experiment, each of these techniques is used to extract a character from a single image, generated by SDXL [57] from an input prompt p. The same prompt p is also provided as input to our method. Textual Inversion (TI) [20] optimizes a textual token using several images of the same concept, and we converted it to support SDXL by learning two text tokens, one for each of its text encoders, as we did in our method. In addition, we used LoRA DreamBooth [71] (LoRA DB), which we found less prone to overfitting than standard DB. Furthermore, we compared against all available image encoder techniques that encode a single image into the textual space of the diffusion model for later generation in novel contexts: BLIP-Diffusion [42], ELITE [89], and IP-adapter [93]. For all the baselines, we used the same prompt p to generate a single image, and used it to extract the identity via optimization (TI and LoRA DB) or encoding (ELITE, BLIP-diffusion and IP-adapter). In Figure 5 we qualitatively compare our method against the above baselines. While TI [20], BLIP-diffusion [42] and IP-adapter [93] are able to follow the specified prompt, they fail to produce a consistent character. LoRA DB [71] succeeds in consistent generation, but it does not always respond to the prompt. Furthermore, the resulting character is generated in the same fixed pose. ELITE [90] struggles with prompt following and the generated characters tend to be deformed. In comparison, our method is able to follow the prompt and maintain consistency, while generating appealing characters in different poses and viewing angles. In order to automatically evaluate our method and the baselines quantitatively, we instructed ChatGPT [53] to generate prompts for characters of different types (e.g., animals, creatures, objects, etc.) in different styles (e.g., stickers, animations, photorealistic images, etc.). Each of these prompts was then used to extract a consistent character by our method and by each of the baselines. Next, we generated these characters in a predefined collection of novel contexts. For a visual comparison, please refer to the supplementary material. We employ two standard evaluation metrics: prompt similarity and identity consistency, which are commonly used in the personalization literature [6, 20, 70]. Prompt similarity measures the correspondence between the generated images and the input text prompt. We use the standard CLIP [61] similarity, i.e., the normalized cosine similarity between the CLIP image embedding of the generated images and the CLIP text embedding of the source prompts. For measuring identity consistency, we calculate the similarity between the CLIP image embeddings of generated images of the same concept across different contexts. As can be seen in Figure 6 (left), there is an inherent trade-off between prompt similarity and identity consistency: LoRA DB and ELITE exhibit high identity consistency, while sacrificing prompt similarity. TI and BLIPdiffusion achieve high prompt similarity but low identity consistency. Our method and IP-adapter both lie on the Pareto front. However, our method achieves better identity consistency than IP-adapter, which is significant from the user’s perspective, as supported by our user study. 4.2. User Study We conducted a user study to evaluate our method, using the Amazon Mechanical Turk (AMT) platform [2]. We used the same generated prompts and samples that were used in Section 4.1 and asked the evaluators to rate the prompt similarity and identity consistency of each result on a Likert scale of 1–5. For ranking the prompt similarity, the evaluators were presented with the target text prompt and the result of our method and the baselines on the same page, and were asked to rate each of the images. For identity consistency, for each of the generated concepts, we compared our method and the baselines by randomly choosing pairs of generated images with different target prompts, and the evaluators were asked to rate on a scale of 1–5 whether the images contain the same main character. Again, all the pairs of the same character for the different baselines were shown on the same page. As can be seen in Figure 6 (right), our method again exhibits a good balance between identity consistency and prompt similarity, with a wider gap separating it from the baselines. For more details and statistical significance analbaselines. For more details and statistical significance analysis, read the supplementary material. 4.3. Ablation Study We conducted an ablation study for the following cases: (1) Without clustering — we omit the clustering step described in Section 3.1, and instead simply generate 5 images according to the input prompt. (2) Without LoRA — we reduce the optimizable representation Θ in the identity extraction stage, as described in Section 3.2, to consist of only the newly-added text tokens without the additional LoRA weights. (3) With re-initialization — instead of using the latest representation Θ in each of the optimization iterations, as described in Section 3.3, we re-initialize it in each iteration. (4) Single iteration — rather than iterating until convergence (Section 3.3), we stop after a single iteration. As can be seen in Figure 6 (left), all of the above key components are crucial for achieving a consistent identity in the final result: (1) removing the clustering harms the identity extraction stage because the training set is too diverse, (2) reducing the representation causes underfitting, as the model does not have enough parameters to properly capture the identity, (3) re-initializing the representation in each iteration, or (4) performing a single iteration, does not allow the model to converge into a single identity. For a visual comparison of the ablation study, as well as comparison of alternative feature extractors (DINOv1 [14] and CLIP [61]), please refer to the supplementary material. 4.4. Applications As demonstrated in Figure 7, our method can be used for various down-stream tasks, such as (a) Illustrating a story by breaking it into a different scenes and using the same consistent character for all of them. (b) Local text-driven image editing by integrating Blended Latent Diffusion [5, 7] — a consistent character can be injected into a specified location of a provided background image, in a novel pose specified by a text prompt. (c) Generating a consistent character with an additional pose control using ControlNet [97]. For more details, please refer to the supplementary material. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

The Chosen One: Consistent Characters in Text-to-Image Diffusion Models: Experiments

Authors:
(1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem;
(2) Amir Hertz, Google Research;
(3) Yael Vinker, Google Research and Tel Aviv University;
(4) Moab Arar, Google Research and Tel Aviv University;
(5) Shlomi Fruchter, Google Research;
(6) Ohad Fried, Reichman University;
(7) Daniel Cohen-Or, Google Research and Tel Aviv University;
(8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Table of Links Abstract and Introduction
Related Work
Method
Experiments
Limitations and Conclusions
A. Additional Experiments
B. Implementation Details
C. Societal Impact
References B. Implementation Details In this section, we provide the implementation details that were omitted from the main paper. In Appendix B.1 we provide the implementation details of our method and the baselines. Then, in Appendix B.2 we provide the implementation details of the automatic metrics that we used to evaluate our method against the baselines. In Appendix B.3 we provide the implementation details and the statistical analysis for the user study we conducted. Lastly, in Appendix B.4 we provide the implementation details for the applications we presented. B.1. Method Implementation Details We based our method, and all the baselines (except ELITE [90] and BLIP-diffusion [42]) on Stable Diffusion XL (SDXL) [57], which is the state-of-the-art open source text-to-image model, at the writing of this paper. We used the official ELITE implementation, that uses Stable Diffusion V1.4, and the official implementation of BLIPdiffusion, that uses Stable Diffusion V1.5. We could not replace these two baselines to SDXL backbone, as the encoders were trained on these specific models. As for the rest of the baselines, we used the same SDXL architecture and weights. List of the third-party packages that we used: • Official SDXL [57] implementation by HuggingFace Diffusers [86] at https : / / github . com / huggingface/diffusers • Official SDXL LoRA DB implementation by HuggingFace Diffusers [86] at https : / / github . com / huggingface/diffusers. • Official ELITE [90] implementation at https : / / github.com/csyxwei/ELITE • Official BLIP-diffusion [42] implementation at https: / / github . com / salesforce / LAVIS / tree / main/projects/blip-diffusion • Official IP-adapter [93] implementation at https:// github.com/tencent-ailab/IP-Adapter • DINOv2 [54] ViT-g/14, DINOv1 [14] ViT-B/16 and CLIP [61] ViT-L/14 implementation by HuggingFace Transformers [91] at https : / / github . com / huggingface/transformers B.2. Automatic Metrics Implementation Details In order to automatically evaluate our method and the baselines quantitatively, we instructed ChatGPT [53] to generate prompts for characters of different types (e.g., animals, creatures, objects, etc.) in different styles (e.g., stickers, animations, photorealistic images, etc.). These prompts were then used to generate a set of consistent characters by our method and by each of the baselines. Next, these prompts were used to generate these characters in a predefined collection of novel contexts from the following list: • “a photo of [v] at the beach” • “a photo of [v] in the jungle” • “a photo of [v] in the snow” • “a photo of [v] in the street” • “a photo of [v] with a city in the background” • “a photo of [v] with a mountain in the background” • “a photo of [v] with the Eiffel Tower in the background” • “a photo of [v] near the Statue of Liberty” • “a photo of [v] near the Sydney Opera House” • “a photo of [v] floating on top of water” • “a photo of [v] eating a burger” • “a photo of [v] drinking a beer” • “a photo of [v] wearing a blue hat” • “a photo of [v] wearing sunglasses” • “a photo of [v] playing with a ball” • “a photo of [v] as a police officer” where [v] is the newly-added token that represents the consistent character. B.3. User Study Details As explained in Section 4.2, we conducted a user study to evaluate our method, using the Amazon Mechanical Turk (AMT) platform [2]. We used the same generated prompts and samples that were used in Section 4.1, and asked the evaluators to rate the prompt similarity and identity consistency of each result on a Likert scale of 1–5. For ranking the prompt similarity, the evaluators were instructed the following: “For each of the following images, please rank on a scale of 1 to 5 its correspondence to this text description: {PROMPT}. The character in the image can be anything (e.g., a person, an animal, a toy etc.” where {PROMPT} is the target text prompt (in which we replaced the special token with the word “character”). All the baselines, as well as our method, were presented in the same page, and the evaluators were asked to rate each one of the results using a slider from 1 (“Do not match at all”) to 5 (“Match perfectly”). Next, to assess identity consistency, we took for each one of the characters two generated images that correspond to different target text prompts, put them next to each other, and instructed the evaluators the following: “For each of the following image pairs, please rank on a scale of 1 to 5 if they contain the same character (1 means that they contain totally different characters and 5 means that they contain exactly the same character). The images can have different backgrounds”. We put all the compared images on the same page, and the evaluators were asked to rate each one of the pairs using a slider from 1 (“Totally different characters”) to 5 (“Exactly the same character”). We collected three ratings per question, resulting in 1104 ratings per task (prompt similarity and identity consistency). The time allotted per task was one hour, to allow the raters to properly evaluate the results without time pressure. The means and variances of the user study responses are reported in Table 1. In addition, we conducted a statistical analysis of our user study by validating that the difference between all the conditions is statistically significant using Kruskal-Wallis [40] test (p < 1e−28 for the text similarity test and p < 1e−76 for the identity consistency text). Lastly, we used Tukey’s honestly significant difference procedure [83] to show that the comparison of our method against all the baselines is statistically significant, as detailed in Table 2. B.4. Applications Implementation Details In Section 4.4, we presented three downstream applications of our method. Story illustration. Given a long story, e.g., “This is a story about Jasper, a cute mink with a brown jacket and red pants. Jasper started his day by jogging on the beach, and afterwards, he enjoyed a coffee meetup with a friend in the heart of New York City. As the day drew to a close, he settled into his cozy apartment to review a paper”, one can create a consistent character from the main character description (“a cute mink with a brown jacket and red pants”), then they can generate the various scenes by simply rephrasing the sentence: “[v] jogging on the beach”


“[v] drinking coffee with his friend in the heart of New York City”


“[v] reviewing a paper in his cozy apartment” Local image editing. Our method can be simply integrated with Blended Latent Diffusion [5, 7] for editing images locally: given a text prompt, we start by running our method to extract a consistent identity, then, given an input image and mask, we can plant the character in the image within the mask boundaries. In addition, we can provide a local text description for the character. Additional pose control. Our method can be integrated with ControlNet [97]: given a text prompt, we first apply our method to extract a consistent identity Θ = (θ, τ ), where θ are the LoRA weights and τ is a set of custom text embeddings. Then, we can take an off-the-shelf pre-trained ControlNet model, plug-in our representation Θ, and use it to generate the consistent character in different poses given by the user. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem; (2) Amir Hertz, Google Research; (3) Yael Vinker, Google Research and Tel Aviv University; (4) Moab Arar, Google Research and Tel Aviv University; (5) Shlomi Fruchter, Google Research; (6) Ohad Fried, Reichman University; (7) Daniel Cohen-Or, Google Research and Tel Aviv University; (8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Authors: Authors: (1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem; (2) Amir Hertz, Google Research; (3) Yael Vinker, Google Research and Tel Aviv University; (4) Moab Arar, Google Research and Tel Aviv University; (5) Shlomi Fruchter, Google Research; (6) Ohad Fried, Reichman University; (7) Daniel Cohen-Or, Google Research and Tel Aviv University; (8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Table of Links Abstract and Introduction Related Work Method Experiments Limitations and Conclusions A. Additional Experiments B. Implementation Details C. Societal Impact References Abstract and Introduction Abstract and Introduction Related Work Related Work Method Method Experiments Experiments Limitations and Conclusions Limitations and Conclusions A. Additional Experiments A. Additional Experiments B. Implementation Details B. Implementation Details C. Societal Impact C. Societal Impact References References B. Implementation Details In this section, we provide the implementation details that were omitted from the main paper. In Appendix B.1 we provide the implementation details of our method and the baselines. Then, in Appendix B.2 we provide the implementation details of the automatic metrics that we used to evaluate our method against the baselines. In Appendix B.3 we provide the implementation details and the statistical analysis for the user study we conducted. Lastly, in Appendix B.4 we provide the implementation details for the applications we presented. B.1. Method Implementation Details We based our method, and all the baselines (except ELITE [90] and BLIP-diffusion [42]) on Stable Diffusion XL (SDXL) [57], which is the state-of-the-art open source text-to-image model, at the writing of this paper. We used the official ELITE implementation, that uses Stable Diffusion V1.4, and the official implementation of BLIPdiffusion, that uses Stable Diffusion V1.5. We could not replace these two baselines to SDXL backbone, as the encoders were trained on these specific models. As for the rest of the baselines, we used the same SDXL architecture and weights. List of the third-party packages that we used: • Official SDXL [57] implementation by HuggingFace Diffusers [86] at https : / / github . com / huggingface/diffusers • Official SDXL LoRA DB implementation by HuggingFace Diffusers [86] at https : / / github . com / huggingface/diffusers. • Official ELITE [90] implementation at https : / / github.com/csyxwei/ELITE • Official BLIP-diffusion [42] implementation at https: / / github . com / salesforce / LAVIS / tree / main/projects/blip-diffusion • Official IP-adapter [93] implementation at https:// github.com/tencent-ailab/IP-Adapter • DINOv2 [54] ViT-g/14, DINOv1 [14] ViT-B/16 and CLIP [61] ViT-L/14 implementation by HuggingFace Transformers [91] at https : / / github . com / huggingface/transformers B.2. Automatic Metrics Implementation Details In order to automatically evaluate our method and the baselines quantitatively, we instructed ChatGPT [53] to generate prompts for characters of different types (e.g., animals, creatures, objects, etc.) in different styles (e.g., stickers, animations, photorealistic images, etc.). These prompts were then used to generate a set of consistent characters by our method and by each of the baselines. Next, these prompts were used to generate these characters in a predefined collection of novel contexts from the following list: • “a photo of [v] at the beach” • “a photo of [v] in the jungle” • “a photo of [v] in the snow” • “a photo of [v] in the street” • “a photo of [v] with a city in the background” • “a photo of [v] with a mountain in the background” • “a photo of [v] with the Eiffel Tower in the background” • “a photo of [v] near the Statue of Liberty” • “a photo of [v] near the Sydney Opera House” • “a photo of [v] floating on top of water” • “a photo of [v] eating a burger” • “a photo of [v] drinking a beer” • “a photo of [v] wearing a blue hat” • “a photo of [v] wearing sunglasses” • “a photo of [v] playing with a ball” • “a photo of [v] as a police officer” where [v] is the newly-added token that represents the consistent character. B.3. User Study Details As explained in Section 4.2, we conducted a user study to evaluate our method, using the Amazon Mechanical Turk (AMT) platform [2]. We used the same generated prompts and samples that were used in Section 4.1, and asked the evaluators to rate the prompt similarity and identity consistency of each result on a Likert scale of 1–5. For ranking the prompt similarity, the evaluators were instructed the following: “For each of the following images, please rank on a scale of 1 to 5 its correspondence to this text description: {PROMPT}. The character in the image can be anything (e.g., a person, an animal, a toy etc.” where {PROMPT} is the target text prompt (in which we replaced the special token with the word “character”). All the baselines, as well as our method, were presented in the same page, and the evaluators were asked to rate each one of the results using a slider from 1 (“Do not match at all”) to 5 (“Match perfectly”). Next, to assess identity consistency, we took for each one of the characters two generated images that correspond to different target text prompts, put them next to each other, and instructed the evaluators the following: “For each of the following image pairs, please rank on a scale of 1 to 5 if they contain the same character (1 means that they contain totally different characters and 5 means that they contain exactly the same character). The images can have different backgrounds”. We put all the compared images on the same page, and the evaluators were asked to rate each one of the pairs using a slider from 1 (“Totally different characters”) to 5 (“Exactly the same character”). We collected three ratings per question, resulting in 1104 ratings per task (prompt similarity and identity consistency). The time allotted per task was one hour, to allow the raters to properly evaluate the results without time pressure. The means and variances of the user study responses are reported in Table 1. In addition, we conducted a statistical analysis of our user study by validating that the difference between all the conditions is statistically significant using Kruskal-Wallis [40] test (p < 1e−28 for the text similarity test and p < 1e−76 for the identity consistency text). Lastly, we used Tukey’s honestly significant difference procedure [83] to show that the comparison of our method against all the baselines is statistically significant, as detailed in Table 2. B.4. Applications Implementation Details In Section 4.4, we presented three downstream applications of our method. Story illustration. Given a long story, e.g., “This is a story about Jasper, a cute mink with a brown jacket and red pants. Jasper started his day by jogging on the beach, and afterwards, he enjoyed a coffee meetup with a friend in the heart of New York City. As the day drew to a close, he settled into his cozy apartment to review a paper” , one can create a consistent character from the main character description (“a cute mink with a brown jacket and red pants”) , then they can generate the various scenes by simply rephrasing the sentence: Story illustration. “This is a story about Jasper, a cute mink with a brown jacket and red pants. Jasper started his day by jogging on the beach, and afterwards, he enjoyed a coffee meetup with a friend in the heart of New York City. As the day drew to a close, he settled into his cozy apartment to review a paper” (“a cute mink with a brown jacket and red pants”) “[v] jogging on the beach” “[v] drinking coffee with his friend in the heart of New York City” “[v] reviewing a paper in his cozy apartment” “[v] jogging on the beach” “[v] jogging on the beach” “[v] jogging on the beach” “[v] drinking coffee with his friend in the heart of New York City” “[v] drinking coffee with his friend in the heart of New York City” “[v] drinking coffee with his friend in the heart of New York City” “[v] reviewing a paper in his cozy apartment” “[v] reviewing a paper in his cozy apartment” “[v] reviewing a paper in his cozy apartment” Local image editing. Our method can be simply integrated with Blended Latent Diffusion [5, 7] for editing images locally: given a text prompt, we start by running our method to extract a consistent identity, then, given an input image and mask, we can plant the character in the image within the mask boundaries. In addition, we can provide a local text description for the character. Local image editing. Additional pose control. Our method can be integrated with ControlNet [97]: given a text prompt, we first apply our method to extract a consistent identity Θ = (θ, τ ), where θ are the LoRA weights and τ is a set of custom text embeddings. Then, we can take an off-the-shelf pre-trained ControlNet model, plug-in our representation Θ, and use it to generate the consistent character in different poses given by the user. Our method can be integrated with ControlNet [97]: given a text prompt, This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Stories, News & Mentions about PHOTONE

Photone’s Stories on HackerNoon