Moving Beyond Animations to User Interactions at 60 FPS in React Native

presentation is everything The async nature of the React Native bridge incurs an inherent performance penalty, preventing JavaScript code from running at high framerates. Modern animation libraries, like Animated, address this by minimizing passes over the bridge. User interactions, where UI continuously reacts to the user’s gestures, are a step further. How can we run those at 60 FPS? Crossing the last mile React Native has a lot of appeal as the stack of choice for modern mobile apps. The major advantage this framework offers is a dramatic increase in . Simply put, you develop apps much faster — partly due to the fact you can finally share code between platforms. productivity There’s always a concern though. Will React Native be able to take me across ? Will the app that I’ll produce measure up to top of the line apps that are implemented in pure native? the last mile I have to admit that this concern is valid. At Wix.com, we switched our mobile stack about a year ago from a purely native one, with separate codebases for iOS and Android, to React Native. The first 95% of development was a breeze. We found ourselves moving forward at about 4 times our previous pace. The last 5% though, was a bit more of a challenge. We found that these 5%, what I call , is still not straightforward to implement with React Native. the last mile It is our goal as community to improve that. So what makes great apps? What are the nuances that set the best apps apart from the mediocre ones? In mobile, we’ve grown to expect that objects no longer just pop up on screen. Things are expected to move around in smooth transitions. Fluid animations at 60 FPS are an important part of the last 5%. Animations used to be a big issue in React Native. This issue was ultimately resolved using , the excellent animation library that’s part of the core. Animated Let’s look at the next step beyond animations — dynamic user interactions that mimic reality. An interaction takes place when the user performs a gesture on a view, and this view continuously responds to the user’s gesture with physical realism. Let’s look at some real life examples to better understand what we’re talking about. I went through my personal phone and started cataloging examples of great interactions in some of my favorite apps: UX Inspirations — On the left we have the official iOS Mail app by Apple and the app from Google. As the user swipes rows in the ListView, buttons for row actions gradually appear from the side. ListView row actions Inbox by Gmail — Second from the left we have the app by Google and the app by Lifehack Labs which has -like UI. As the user is swiping, these cards modify their appearance and if swiped with enough force, fly off screen. Swipeable cards Google Now Flic Tinder — Second from the right we have and the app by Any.DO. Both of these have views that the user can collapse between multiple states. Switching between filter and search in Airbnb and switching between month-view to week-view in Cal. Collapsible views Airbnb Cal — On the right we have the official iOS top notification panel by Apple and the official iOS Maps app by Apple. The user can drag these panels to reveal additional UI elements that are normally hidden. Much like the popular navigation drawer / side menu. Sliding panels & drawers What do these examples have in common? They are all physical in nature. The views have velocities that are changing as they’re being dragged and tossed. Notice the nuances, like how the notification panel bounces from the ground when thrown with enough force. Implementation with JavaScript When using React Native, we would naturally try to implement these interactions in JavaScript. Let’s review such an implementation. The first inspiration example — ListView row actions — is actually implemented in React Native core under the name . SwipeableRow It has a modern implementation with all the latest and greatest. It puts emphasis on performance and makes heavy use of the Animated library. Let’s focus our attention on the part that implements the interaction itself: The implementation relies on a to calculate the changes to views between touch events. What performance should we expect from this approach? PanResponder To analyze performance, we’ll have to look into React Native internals. React Native has two realms running side by side: the — where we implement our business logic, and the — where our native views reside. Communication between the two realms takes place over . Since serialization is required to send data over the bridge, frequent communication is expensive. JavaScript realm native realm the bridge Touch events are a native construct, they originate in the . For every frame of the interaction, these events are sent over the bridge to be handled by in the . Once the business logic calculates the response, an Animated Value is set. Since updating views has to take place in the , we have to cross the bridge one more time. native realm _handlePanResponderMove JavaScript realm native realm As you can see, every frame requires data to be serialized over the bridge. If your app is busy, you’ll find that this performance overhead will prevent the interaction from running at 60 FPS. Implementation with native While working on the Wix app, we originally started implementing all interactions with JavaScript. When performance wasn’t as smooth as expected, we started porting specific use-cases to native. This meant implementing everything twice — once in Objective-C for iOS and once in Java for Android. It’s usually easier to reach 60 FPS with a native implementation because we can avoid passing data over the bridge and close the entire loop, business logic and views both, in the . native realm Since we open source almost all of our native code, we’ve ended up with multiple libraries like that implements swipeable cards and that implements swipeable row actions. Without a general purpose solution, every new use-case results in yet another custom-tailored library. react-native-swipe-view react-native-action-view The main problem with this approach is that it requires native skillset and usually two different developers. At Wix, we maintain about 10% of our frontend workforce as native engineers with expertise in Objective-C/Swift or Java for this purpose. This is not good enough. We should aim higher and try to find an elegant general purpose solution. Learning from animations Animations actually present a very similar challenge. The naive implementation would tween view properties between frames in JavaScript. This would generate a lot of noise over the bridge and result in frame loss. As we know, the library emerged as the solution to deal with animations at 60 FPS in React Native. How does it work? Animated The concept behind Animated is using a to describe animations. If we are able to declare the entire animation in advance, then the declaration in JavaScript can be serialized and sent over the bridge once. From that point on, a will execute the animation frame by frame according to the spec in the declaration. declarative API general purpose driver The original driver for Animated was implemented in JavaScript. The recent , though, provide a that is able to execute the animation frame by frame in the and update the native views without going over the bridge. versions native driver native realm This method reduces traffic over the bridge to the initialization phase only. This brings us to an interesting conclusion: Declarative API is how we cross the last mile This is a very powerful concept. These are the sort of libraries we should be thinking about. Whenever we find a performance boundary in React Native, this is a way to push against it. All we have to do is find several example use-cases and design a declarative API that can cover all of them. That’s exactly what we’re going to do next. Declarative API for user interactions In order to design a successful API we should define a couple of goals: Our API should be general purpose. A good way to verify that is make sure it covers all 8 examples we saw in our UX inspirations above. Our API should be simple. A good way to verify that is make sure every interaction takes no more than 3–5 lines of code to define. Before we jump into the specifics of our API, I want to mention some very interesting work going on in aimed at adding some support for user interactions. One interesting addition is which allows to perform view property interpolations based on position. Another interesting work in-progress is a library by named which allows to perform view property interpolations based on gesture parameters. Animated Animated.ScrollView ScrollView Krzysztof Magiera react-native-gesture-handler The approach we’re going to take together now is a little different. We’re going to start from the 8 UX inspirations shown above and design the simplest high-level API that can define all of them. Defining the API — phase 1 Analyzing our 8 , we can see that some of the views are free to move horizontally and some are free to move vertically. Therefore, specifying the is a good start for our API. UX inspirations direction Another observation is that the views are only free to move while dragged. Once the user lets go, they usually snap to one of predefined . Drawers, for example, snap either to an open position or a closed position. snap points Lastly, to give the snap behavior a realistic feel, we need to use something like a spring animation curve. If we don’t want the spring to oscillate forever, we should also specify in our API the (or damping of the spring). friction In summary, the first phase of our declarative API can rely on the props: / horizontal vertical snap points friction Let’s try to use this simple API to declare the first two UX inspirations — ListView row actions (left) and swipeable cards (right): In order to allow the swipeable cards to be swiped away, we simply define snap points that are completely off-screen (-360 and 360 logical pixels). Note that we currently use pixel values for simplicity. We can add support later for units that are better suited for multiple screen resolutions — such as percentages. This is a great start, but designing the is only the first half. The second half is implementing the . Let’s do that next. declarative API native driver Implementing the native driver — attempt 1 After the specs of the interaction have been declared as props in the , they are serialized by React Native during initialization and sent over the bridge once to the . Our general purpose will receive these specs and drive the interaction entirely from native. There will be no more passes over the bridge required to calculate each frame, resulting in overhead-free execution at 60 FPS. JavaScript realm native realm native driver Let’s start with a simple implementation in Objective-C. We’ll drag the view by using a and when the pan gesture ends, we’ll find the closest snap point and animate our view to it with a spring curve: UIPanGestureRecognizer This implementation works well enough. The problem with it is that we’re faking the physics with animation. Consider what happens when the view is tossed by the user at some initial velocity. The animation function we’re using can only apply velocity in the direction of the spring. What happens if the user tosses the view in another direction? Our model isn’t powerful enough to drive this case. Implementing the native driver — attempt 2 Let’s look into more powerful models to drive the interaction. If you dive into the native SDKs and check how Apple recommends implementing complex interactions with physical realism, you’ll run into . UIKit Dynamics This crazy API was introduced in iOS 7. It runs a full fledged physics engine under the hood and allows us to apply physical properties like , and to views. The physical parameters of the scene are defined by applying . We can easily modify the implementation above: mass velocity forces behaviors We’re getting closer, but we’re still not there. Using UIKit Dynamics has two major drawbacks. First, there’s no Android support. This API is exclusive to iOS and there’s no parallel in the Android SDK. Second, some behaviors, such as , don’t provide enough control — there’s no way to specify the strength of the snapping force for example. snap Implementing the native driver — attempt 3 Let’s get a little bit crazier. Why not try to implement UIKit Dynamics by ourselves? At the end of the day, the physical forces are relatively simple mathematical equations. Building the physics engine from scratch shouldn’t be too difficult. UIKit Dynamics will show us the way. We can even adopt its behavior pattern. Let’s take the snap behavior for example — we can implement it using a . How does a spring behave? Time to recall some : spring Physics 101 Don’t worry about the math too much, this is something the library will do internally. The Wikipedia entries for and can provide you with the full background. Newton’s laws of motion Hooke’s law We’ll have to calculate the forces and velocities on every frame. To set this up, we’ll need a high precision timer running at 60 FPS. Luckily, there’s native API designed specifically for this task — . Putting it all together will yield the following: CADisplayLink Now this feels right, and brings us to a very interesting realisation… We’re writing a declarative physics engine for React Native And this is pretty damn cool. We finally have the under control. It’s time to make use of our powerful engine and add some more capabilities to our . native driver declarative API Enriching the API — more props The declarative API we have so far provides a solid foundation but still lacks ability to implement some of the more intricate interactions in our 8 UX inspirations. Consider the official iOS top notification panel by Apple. When the user throws the panel with enough force downwards, the panel off the ground. bounces We can easily add support for this behavior to our declarative API. We’ll limit the view’s movement with and add from the edges: boundaries bounce Let’s consider another complex use-case, this time involving ListView row actions. Some rows don’t have action buttons on both sides. When this is the case, the common UX behavior is to allow the row to move freely in the direction exposing the buttons, but when moved in the other direction, movement will be more difficult and meet increasing resistance. We can add resistance to the row movement by tying one of its edges to the screen’s edge using a constant . Unlike snap points, this spring will also be active while dragging. spring We still need to sort out another issue. The row should move left without resistance (this direction exposes buttons) but move right with resistance (the direction without buttons). We can add this behavior to our API by giving every force, such as our spring, an optional . influence area When the view is outside the influence area, the force will disappear. As you can see, as we meet more and more use-cases we can simply enrich our declarative API and add general purpose abilities to describe them. Enriching the API — integration with Animated We’re still missing a large piece of the puzzle. Consider the ListView row actions use-case. As you swipe the row, the action buttons gradually appear from beneath. A common pattern is to change their appearance, like scale and opacity, as they are revealed. You can see this behavior below (the action buttons in blue): Also note that the views that we want to animate (the blue action buttons) are different from the view the user is interacting with (the gray row cover). This effect isn’t trivial to implement because the stage of the animation depends on the horizontal position of the row instead of timing. Nevertheless, this is still an animation — where view properties (scale and opacity) are modified in sequence. We already have a powerful tool for animating view properties at our disposal — the library. Let’s find a way to use it for our purposes. Animated View property animations with Animated are performed declaratively by defining interpolations over an : Animated.Value Since the animation depends on the horizontal position of the row, what if we transmit the position into an Animated.Value? This will allow us to define interpolations based on the interactable view’s position that affect other views that aren’t a direct part of the interaction (like the buttons). How would that work in our declarative API? We can specify this behavior by passing the Animated.Value as a prop ( ): animatedValueX Our native driver will perform the actual transmission under the hood. This can be accomplished by using . Recent versions of Animated even support driving Animated.events using a . This means that the entire animation — from transmitting the position to interpolating and updating the view properties — can be performed in the without sending data over the bridge. This is great news if we’re aiming at 60 FPS. Animated.events native driver native realm Enriching the API — finishing touches If we’re doing our own physics, we might as well add the rest of the forces. We already have , let’s add and too. This will provide developers with the flexibility needed to define all sorts of crazy physical interactions. springs gravity magnetism We should also add support for , so our JavaScript code could be notified when an interaction stopped or when the view snapped to a point. And while we’re at it, adding is also a nice touch — so the device will vibrate slightly whenever a view collides with its surroundings. These nuances add the polish required for a great user experience. events haptic feedback Time to wrap things up… I want to show you the full power of what we’ve created here. Take a look at the following declaration, can you guess what it implements? Our mystery view snaps either to the left or right edges of the screen. There’s a gravity well in the bottom, which sucks the view inside when it gets too close. Also, notice that we don’t limit movement and allow the view to move in both directions. What we have here is a full implementation in 7 lines of code! chat heads Does it really run at 60 FPS? Seeing the videos isn’t the same as experiencing the interactions by yourself on a real device. Note that even the simulator doesn’t provide the real experience as it drops frames. So, on a real device, does it really run at 60 FPS? Judge for yourself. I’ve implemented all 8 with the engine we’ve just created using our declarative API in React Native. You can find the resulting demo app on the (iOS) and (Android). UX inspirations Apple App Store Google Play Download the demo app The full implementations of the physics engine, our native driver for iOS and Android, and the demo app are available on GitHub: _react-native-interactable - Experimental implementation of high performance interactable views in React Native_github.com wix/react-native-interactable Special thanks to and Tzachi Kopylovitz for helping bring this home in time for ReactConf 2017. Rotem Mizrachi-Meidan Crossing the last mile I hope you’ve taken from this interesting experiment more than just a cool way to implement great user interactions in React Native. Our goal as a community is to identify the boundaries of React Native and then to push against them. When you stumble upon an interesting performance problem in React Native, I urge you to find several example use-cases and try to design a simple declarative API that can define them. If the performance problem stems from bridge overhead (as is usually the case), a native driver for your API will probably provide a good solution. Let’s cross the last mile together.