Talkpush

talkpush

Film Trailer Generation via Task Decomposition: Conclusions and References

টাস্ক ডিকম্পোজিশনের মাধ্যমে ফিল্ম ট্রেলার জেনারেশন: উপসংহার এবং রেফারেন্স

Filmtrailer-Generierung durch Aufgabenzerlegung: Schlussfolgerungen und Referenzen

Generación de avances de películas mediante descomposición de tareas: conclusiones y referencias

Génération de bandes-annonces de films via la décomposition de tâches : conclusions et références

टास्क डिकंपोज़िशन के माध्यम से फ़िल्म ट्रेलर निर्माण: निष्कर्ष और संदर्भ

タスク分解による映画予告編生成: 結論と参考文献

작업 분해를 통한 영화 예고편 생성: 결론 및 참고 자료

Geração de trailer de filme por meio de decomposição de tarefas: conclusões e referências

Генерация трейлеров к фильму посредством декомпозиции задач: выводы и ссылки

Görev Ayrıştırma Yoluyla Film Fragmanı Oluşturma: Sonuçlar ve Referanslar

Tạo đoạn giới thiệu phim thông qua phân tách nhiệm vụ: Kết luận và tài liệu tham khảo

通过任务分解生成电影预告片：结论与参考文献

Hey friends! Have you ever wanted to create your own custom indicators for TradingView? Well, you're in luck - in this post, I'll be walking you through a step-by-step guide on how to write a basic indicator from scratch using PineScript. For those who don't know, is a popular online charting platform used by traders to analyze financial markets. It allows you to create charts with various indicators and drawing tools and also features a huge community of traders who publish ideas and custom indicators. TradingView is the built-in coding language used on TradingView to program your own indicators and strategies. It's pretty straightforward to learn, especially if you have any coding experience. I'll be focusing on PineScript version 5 which is the latest iteration with new features and improvements. PineScript So let's get started and get familiar with PineScript syntax, code your first basic indicator, and publish it for the community - by the end, you'll have the skills to bring any custom indicator idea to life Setting Up Your TradingView Account The first step is to head over to Tradingview and . The free plan is enough to follow this tutorial and publish an indicator. sign up for a free account Once you log in, you'll see the TradingView charting homepage. The Chart tab is where you'll spend time analyzing price charts from global markets. The Pine Editor is where we'll code the indicator script. With your account created and the platform explored, you now have everything needed to start coding a custom indicator. Time to move on to learning PineScript! Introducing PineScript PineScript is a specific to TradingView that lets you create trading algorithms, indicators, strategies, and more. It uses a JavaScript-like syntax to build scripts that run natively in the TradingView charting environment. programming language Some key facts about PineScript: Used only on TradingView, interpreted and executed by their servers Built-in variables and functions tailored for financial data analysis PineScript v5 is the latest version with new features Open-source with a large community contributing scripts To access the Pine Code Editor, click on the Pine Editor tab at the top. This will open up a blank editor panel where you can start coding your indicator. Think of this as your workbench to build custom PineScript programs. When you first open the editor, you'll see a sample script with some basic code to show variable assignments, a plot function, etc. You can leave it for reference or delete it and start fresh. Now before we jump into writing code, let's get familiar with some PineScript basics. Understanding the Basics of PineScript PineScript shares similarities with languages like JavaScript and C in both syntax and functionality. Let's go over some key concepts and components that we'll need for our first indicator: Series A series in Pine Script refers to sequential data like OHLC prices, indicator values, trade volumes etc. over a timeframe. It is the foundation for accessing and manipulating data in your scripts. Core price series Pine Script has four built-in series for OHLC data: open - Open price of each bar high - High price of each bar low - Low price of each bar close - Close price of each bar For example: // Access close of current bar
currentClose = close

// Access open two bars ago
pastOpen = open[2] Declaring custom series You can declare your own series to store custom indicator values: // Declare series 
var mySeries = na

// Assign values
mySeries := close Accessing prior values Use array index notation to access historical values in a series: // Close 1 bar ago
prevClose = close[1] 

// Close 5 bars ago
pastClose = close[5] Series lifespan Series only exist for visible bars on the chart. If you scroll left or right, the series values are recalculated. Common operations You can perform math, combine series, find minimum/maximum etc: // Add two series
sum = series1 + series2

// Higher value of two series
hi = math.max(series1, series2)

// Moving average of series
sma = sma(mySeries, 20) Data Types Explicitly declare variables with types: // Integer
var int myInteger = 10

// Float
var float myFloat = 10.5 

// String 
var string myString = "Hello"

// Boolean
var bool myBool = true Operators Perform actions like arithmetic, comparisons, etc: // Arithmetic
int sum = 10 + 15
int diff = 20 - 7
int product = 2 * 10

// Comparison 
if close > open
    // Do something

// Logical  
if (trend == "up") and (momentum > 100)
    // Do something Functions Reusable blocks of code to execute specific tasks: // Built-in function
ema20 = ta.ema(close, 20)

// Custom function
foo() => 
    int sum = 10 + 15
    sum

// Call function
value = foo() Control Flow Control execution path based on conditional logic: // If/else statement
if close > open
    // Bull candle
else 
    // Bear candle

// For loop  
for i=0 to 10
    // Do something

// While loop
while close > open
    // Uptrend That covers some of the key concepts. Don't worry if the syntax seems strange - it will start to feel natural with examples. Now let's shift gears and actually build out an indicator! Building Your First Indicator Now that we have covered the basics of Pine Script, let's shift gears and walk through building a practical trading indicator from scratch. In this example, we will code a (RSI) indicator. The RSI is a popular momentum oscillator used by traders to identify overbought and oversold conditions. Relative Strength Index Here is an overview of how RSI works: Calculates the speed and rate of price changes Fluctuates between 0 and 100 Values over 70 indicate an overbought state Values under 30 indicate oversold state Used to detect reversals or divergence Let's break down the step-by-step process: Step 1 - Declare A Script First, let's define a version of PineScript to V5 and then name out indication "RSI". //@version=5
indicator("RSI", overlay=false) indicates that we don't intend for this indicator to overlay the chart. overlay=true Step 2 - Define User Inputs We want to allow the user to customize the RSI period length. We use the input() function: // RSI Period Input
period = input(title="RSI Period", type=input.integer, defval=14) The input() function creates a configurable setting the user can change. Step 3 - Declare RSI Variable We need a variable to store the RSI value on each bar: // RSI series 
var rsi = na The na value initializes it to not applicable before the first value. Step 4 - Calculate RSI in Function In a separate function, we code the RSI calculation logic: // RSI calculation
rsi(period) =>

    up = ta.rma(math.max(ta.change(close), 0), period)
    down = ta.rma(-math.min(ta.change(close), 0), period)
    
    rsi := down == 0 ? 100 : up == 0 ? 0 : 100 - (100 / (1 + up / down)) This uses the change() and rma() functions to calculate the relative strength. Step 5 - Plot RSI on Chart Next we visualize the RSI as a line on the chart: // Plot 
plot(rsi, title="RSI", style=plot.style_line, linewidth=2, color=#FF7200) We style it with a thicker line in orange color. Step 6 - Add Overbought/Oversold Levels Finally, we draw horizontal lines at 70 and 30 for overbought/oversold reference: // Upper band
h1 = hline(70, "Upper Band", color=#C0C0C0)

// Lower band
h2 = hline(30, "Lower Band", color=#C0C0C0) And that's it! We now have a complete RSI indicator coded from scratch in Pine Script. Obviously, this was a very simple example just to demonstrate the basics. You can enhance it further with more configuration options, optimization, alerts, etc. But you should now have a sense of how to go from idea to coded indicator in PineScript. Here are two ways we could add more features to the RSI indicator example using tables and alerts: 1. Create RSI Values Table We can store the RSI values in a table for easy reference: // Define RSI table
table rsiTable(symbol, rsi) 

// On each bar record RSI
table.push(rsiTable, symbol, rsi) Now the user can see all the exact RSI numbers in the data window rather than just the chart line. 2. Add Overbought/Oversold Alerts We can trigger alerts when RSI crosses above 70 or below 30: // Overbought alert
if rsi > 70
    alert("RSI Overbought!")
    
// Oversold alert  
if rsi < 30
    alert("RSI Oversold!") This allows traders to get real-time notifications when the indicator signals important overbought or oversold conditions. The table provides historical data access and alerts to create actionable signals - both useful for traders using the RSI indicator. There are many possibilities like these to expand indicators beyond just visualization on the chart. Here is the complete code! //@version=5
indicator("RSI", overlay=false)

// User Input
period = input(title="RSI Period", type=input.integer, defval=14)

// RSI Series
var rsi = na

// RSI Calculation
rsi(period) =>

    up = ta.rma(math.max(ta.change(close), 0), period)
    down = ta.rma(-math.min(ta.change(close), 0), period)
    
    rsi := down == 0 ? 100 : up == 0 ? 0 : 100 - (100 / (1 + up / down)) 

// Plot RSI	
plot(rsi, title="RSI", style=plot.style_line, linewidth=2, color=#FF7200)

// Overbought Level
h1 = hline(70, "Upper Band", color=#C0C0C0)

// Oversold Level
h2 = hline(30, "Lower Band", color=#C0C0C0)

// RSI Table
table rsiTable(symbol, rsi)
table.push(rsiTable, symbol, rsi)

// Alerts
if rsi > 70
    alert("RSI Overbought!")
    
if rsi < 30
    alert("RSI Oversold!") Next, let's walk through testing your script and publishing it for the TradingView community. Configuring and Testing the Indicator Before our EMA indicator is ready to publish, we need to configure it and test it works as expected. TradingView makes this easy through their charting environment: 1. Add basic settings Let's allow customization of the EMA line color and thickness: // Color setting
col = input(title="Color", type=input.color, defval=#2962FF)

// Thickness 
thm = input(title="Thickness", type=input.integer, defval=2)

// Update plot function
plot(ema_values, "EMA", color=col, linewidth=thm) 2. Add script to a chart In the Pine Editor, click Add to Chart at the bottom. This will load the indicator on a price chart. 3. Observe indicator values Change the symbol and timeframe. Ensure the EMA line plots as expected. Tweak the length and smoothing inputs and see the effect. To inspect the actual EMA values being calculated, you can use the plotchar() function to print values on the chart: // Print EMA values on chart
plotchar(ema_values, "EMA", "▲", location.bottom) The data window also provides the ability to see all indicator values for debugging. Open the data window and look for your "EMA" series. 4. Debug issues If the indicator prints any errors or behaves strangely, go back and debug the script. Fix any typos, logic errors, etc. Checking the printed EMA values compared to the chart can help identify and resolve calculation problems. The data window lets you inspect the exact EMA numbers at each bar to ensure accuracy. Publishing and Sharing the Indicator TradingView makes it simple to save your PineScript creation and share it on their public indicator library: 1. Save script In the Pine Editor, click Save As. Give your indicator a descriptive name and save it. 2. Set visibility to public In the indicator settings, change Visibility to Public. 3. Add description Provide an explanation of what your indicator does in the Description field. 4. Get community feedback Publish your indicator and encourage other users to try it out! Observe their ratings and feedback to improve it over time. And that's all it takes to unleash your TradingView indicator with the world! PineScript has a supportive community, so leverage resources like the PineCoders chat room and wiki to continue enhancing your creation. Conclusion If you've made it this far - congratulations! You now have the skills to code a custom indicator from scratch using PineScript. Of course, this was just a quick overview to get you started. There's a whole lot more you can explore with PineScript like trading strategies, visualizations, integrations, and more. I hope this tutorial gave you a solid foundation to unleash your creativity! So now it's your turn - open up that Pine Editor, get coding, and bring your unique indicator ideas to life. The PineScript community can't wait to see and use what you build. Feel free to reach out if you have any other questions. Happy coding! If you like my article, feel free to follow me on . HackerNoon The lead image for this article was generated by via the prompt "Illustrate arbitrary financial charts" HackerNoon's AI Image Generator

PineScript for Beginners: Writing Your First TradingView Indicator

https://youtu.be/itAbaDoPSwM?embedable=true I'm genuinely thankful every time I get a new listener for the . To get to me, they had to turn down hundreds of other valuable shows that also deserve to be heard or watched. Success Story Podcast There are now over five million podcasts across the major providers, with more than 70 million episodes. What used to be referred to as “alternative media” really isn’t anymore. It has become the mainstream, even if it refuses to accept the moniker. Now it seems like every brand in the world is trying subversive marketing strategies, which pretty much removes the whole point. A couple of years ago, I had a conversation with Andrew Frawley. He was kind of the king of alternative marketing methods for a time. Without much traditional experience, he networked his way into becoming the Chief Marketing Officer for a presidential campaign. ? The internet sensation that went from essentially no national recognition to a household name seemingly overnight? That’s the work of Frawley, who played every trick in the book (and wrote a few chapters himself) to get people talking about his candidate. Remember Andrew Yang But something that Frawley said in that interview has stuck with me ever since. “Alternative media is saturated.” He was right then, even if we hadn’t noticed yet. And it has only gotten worse in the time since. The Clubhouse Saga Launched in 2020 (well, rebranded from Talkshow), Clubhouse was a that (literally) had everyone talking. It exploded during the pandemic's early months, and a year after it launched, the app already had more than ten million active users. social media startup The problem was that every company saw this meteoric rise as an opportunity. Every job posting for a marketing position had a note about emerging platforms like Clubhouse, and users started being constantly bombarded by sponsored content and advertisement. That healthy user base took a huge dip and experienced massive fluctuations over the next year. This spring, Clubhouse announced it was —a move they called “absolutely necessary—and “resetting” the company. laying off 50% of its staff Founder Paul Davison explained that they “grew way, way too fast,” but there was something else wrong with Clubhouse, at least from my perspective. Very quickly, it lost its factor. Lost its . Lost the in alternative media. cool edge alternative Reddit Goes Public The history of Reddit is interesting. It went from a simple link aggregator to where niche enthusiasts would congregate to discuss the latest developments in whatever they were interested in. It had incredibly-engaged communities that could silo themselves away from the rest of the site, enjoying their particular subreddit in peace. But as it embraced the slogan “the front page of the internet,” it eventually became Times Square. There’s lots to do, and you’ll want to be part of it—but I’ll be damned if you don’t see massive flashing billboards in the process. Not to say it is dying. There is still a lot of value in what Reddit brings, and the owners know it. That’s why they’re taking it public this year, or . at least plan to Remember that factor? Yeah, Reddit lost that a long time ago, too. It’s no longer an alternative conduit to an audience you can’t reach with traditional methods. If you want to use it for marketing purposes, you’ll be fighting just about every other major brand in the world for eyeballs. cool So, What’s Next? The question now is simple. If these platforms are so saturated, are they even worth it? Of course they are. I’m not here to tell you to pull your Instagram budget or cancel the podcast you’ve been producing. But you’re kidding yourself if you still consider them “alternative.” What I will suggest is to consider some old ways to do new marketing. Check Your Mailbox: The Unexpected Boon In the digital marketing sphere, action taken by several browsers ( ) to phase out third-party tracking cookies has sent marketers scrambling for new ways to reach their audience. soon including Chrome These cookies, which have long informed personalized digital ad campaigns, are gradually becoming obsolete, leaving marketers grappling with an impending 'cookie apocalypse.' But wait, what's that in your mailbox? A piece of direct mail—tangible, personalized, and immune to the changes in your browser's privacy policies. This 'old-school' channel is proving to be fresher than ever. When the pandemic confined us to our houses, we began spending more time with our mail. An unanticipated consequence of this was . increased engagement with direct mail marketing According to research, direct mail has a better response rate than its digital counterparts, thanks to its personal touch, nostalgic value, and breaks from screen time. Don’t get tied up in the misconception that direct mail only works to create awareness at the top of the funnel. It’s here to influence the entire spectrum, even assisting the bottom of the funnel with critical purchasing decisions. When the recipient is already familiar with the brand, a well-designed piece of direct mail can be the nudge needed to convert consideration into conversion. The icing on the cake? over the years to support targeted digital advertisement can now guide intelligent, laser-focused direct mail campaigns that deliver results. You might lose cookies, but you gain a strong, relevant, and tailored strategy that marries the best of the old and new. The reams of data aggregated Trade Shows: The Human Connection It's easy to forget the power of a good old-fashioned human connection. As we became more remote and distributed in our work, the value of in-person interaction became increasingly evident. There were times over the past few years when I went a whole day (or longer) without speaking to someone face-to-face. I hungered for that interaction, even if it was just with the barista at Starbucks or the cashier at the grocery store. This is how trade shows and conventions, once considered relics of a bygone era, can make an emphatic comeback. Despite the convenience and accessibility of virtual networking, it just doesn't replicate the magic of face-to-face communication. The intangible elements—a firm handshake, a shared laugh, a moment of eye-to-eye connection, an immediate response, and the excitement and energy of a bustling venue—create a level of engagement that any online platform struggles to emulate. The resurgence of trade shows in the age of remote work is a testament to this. It builds trust, connection, and, ultimately, business. An attendee could become a valuable lead, a potential customer, or a long-term business partner, thanks to the quality time and interaction these events provide. In a world where everything seems to be shifting online, this comeback underlines an essential truth: while digital may provide the platform, it's a human relationship that fuels the market. So, whether it's a grand industry expo or a niche event, trade shows are more relevant now than ever. Referrals: Bridging the Trust Gap When asked, business leaders estimate that 87% of consumers trust their organizations, but the reality is starkly different. A meager having confidence in companies. 30% of consumers report Data privacy, shifting economic trends, and divisive social topics have left a lot of customers believing that every company is out to get them or take advantage in some way. Ads, even personalized ones, lose some effectiveness if there is built-in distrust. The only way to reach that target audience is to find something (or someone) that they rely on and respect to deliver the message. Enter referrals and word-of-mouth, the oldest form of marketing. Think about it; doesn't a recommendation from someone you trust carry more weight than any ad, no matter how convincingly presented? Referral marketing leverages this underutilized power of peer-to-peer recommendations. , accelerates the customer journey from consideration to purchase, improves retention rates, and strengthens customer relationships. It builds trust organically A trusted referral is a powerful endorsement, and a single referred customer could become a loyal brand advocate, furthering the cycle. Final Thoughts I’m not trying to say social media marketing is dead. I don’t want you to think that content creation is useless. But you have to understand that these are no longer cutting-edge strategies. Everyone is doing it, and it’ll only get harder as customers become more discerning or technology becomes more sophisticated. The traditional methods aren’t all bad. Sure, newspaper circulation isn’t what it used to be, but not everything from the past should be dismissed so easily. Alternative media isn’t a blue ocean—it’s bright red. If you enjoyed this article, I’d love to hear from you. Write me at or at me , and I’ll do my best to get back to everyone! scott@scottdclary.com tweet @ScottDClary Also published . here

The Saturation of Subculture: How Alternative Media Has Lost Its Edge

Web Scraping with Javascript and Node.js

This article will cover the and how you can use them in your day-to-day JavaScript programming. This article assumes you have some familiarity with coding in the JavaScript ecosystem. spread operator Prerequisites We will be using a code example from a previous post - . Since the spread operator does the opposite thing of what do, a good understanding of would be helpful here. However, as a brief introduction, using the three periods syntax , rest parameters allow us to capture an indefinite number of function arguments as an array. It is kind of similar to or . How to Use Rest Parameters in JavaScript rest parameters rest parameters (…) Java Varargs C# parameter arrays In this code example, the log function takes the and parameters then outputs them to the console. level args { ( i = ; i < args.length; i++) { .log(level, , args[i]);
  }
} // ------------------ 👇 a rest parameter ( ) function log level, ...args for var 0 console `{$i}` The Spread Operator or Spread Syntax Basically, the spread operator converts an array to a list of comma-separated values (CSV), contrasting rest parameters, where it takes a CSV and converts it into an array. We have below a logging function similar to the one above (but without using rest parameters). Let us assume that the variable value comes from an external API that returns a list with varying types and values, and we want to log each element to the console. We can see from the output that while the function is expecting four arguments, we’ve only given it two. Hence the being shown twice for variables and . responseBody undefined arg2 arg3 { .log(level, , arg1); .log(level, , arg2); .log(level, , arg3);
} responseBody = [
  { : }, , ];

log( , responseBody); ( ) function log level, arg1, arg2, arg3 console '{0}' console '{1}' console '{2}' var name 'Non-Master Anakin Skywalker' 1 false 'DEBUG' // expected output: // DEBUG {0} [ { name: 'Non-Master Anakin Skywalker' }, 1, false ] // DEBUG {1} undefined // DEBUG {2} undefined This can be resolved by providing the correct arguments to the function. This can be done by getting each element by index and pass each to the function. log( , responseBody[ ], responseBody[ ], responseBody[ ]); 'DEBUG' 0 1 2 // expected output: // DEBUG {0} { name: 'Non-Master Anakin Skywalker' } // DEBUG {1} 1 // DEBUG {2} false That should be good, our logging function is working as expected, but this can be improved. The desired output can be done with less code using the . We don’t need to access each list element by index. We can use the three periods syntax again here and prepend it to the list variable. Converting it to a comma-separated values that the logging function is expecting. spread operator (…) log( , ...responseBody); // --------- 👇 spread operator 'DEBUG' // expected output: // DEBUG {0} { name: 'Non-Master Anakin Skywalker' } // DEBUG {1} 1 // DEBUG {2} false What was done by using the spread syntax was taking a list and converting it to a CSV. When we used the logging function at first, we missed providing the remaining two arguments. As if we were calling the function like this: log( , [{ : }, , ], , ); 'DEBUG' name 'Non-Master Anakin Skywalker' 1 false undefined undefined Using the spread operator, we have correctly provided the logging function’s arguments. the value of the list into an actual CSV that is accepted by the function. As if we were calling the function like this, which is the correct usage: Spreading log( , { : }, , ); 'DEBUG' name 'Non-Master Anakin Skywalker' 1 false Spread Operator and Rest Parameters If you’re already comfortable with the concept of the spread operator, let us go back to my previous point where it is conceptually the opposite of rest parameters. Let’s use the first logging function from above that uses and use it together with the spread operator. rest parameters When directly passing the list variable, it shows only one debug output to the console. This is expected behavior since what rest parameters do is capture an indefinite number of arguments supplied to a function. In this case, we’ve only given it one argument, thus logging only once. { ( i = ; i < args.length; i++) { .log(level, , args[i]);
  }
} responseBody = [
  { : }, , ];

log( , responseBody); ( ) function log level, ...args for var 0 console `{ }` ${i} var name 'Non-Master Anakin Skywalker' 1 false 'DEBUG' // expected output: // DEBUG {0} [{ name: 'Non-Master Anakin Skywalker' }, 1, false] When passing the list variable using the spread syntax to the function, it shows three debug console outputs, which is the expected output, matching the number of elements from the list. log( , ...responseBody); 'DEBUG' // expected output: // DEBUG {0} { name: 'Non-Master Anakin Skywalker' } // DEBUG {1} 1 // DEBUG {2} false This shows how rest parameters and the spread operator are opposite of each other and how the two can be used together. Let us look at another example of the spread operator but with a list with four elements. This time it will output to the console four times. responseBody = [
  { : }, , ,
  [ , ]
];

log( , ...responseBody); var name 'Non-Master Anakin Skywalker' 1 false 'Tatooine' 'Coruscant' 'DEBUG' // expected output: // DEBUG {0} { name: 'Non-Master Anakin Skywalker' } // DEBUG {1} 1 // DEBUG {2} false // DEBUG {3} [ 'Tatooine', 'Coruscant' ] Adding Element to a List There is also another use for the spread operator aside from passing them as function arguments. Here we have a list variable that has two elements. If we want to add a new element at the end of the list, that should be straightforward. tail head = { : }; tail = [{ : }, { : }]; .log(tail); tail.push(head); .log(tail); const value 'A' const value 'B' value 'C' console // expected output: // [ { value: 'B' }, { value: 'C' } ] console // expected output: // [ { value: 'B' }, { value: 'C' }, { value: 'A' } ] When adding a new element at the front of the list, we can use the spread operator. head = { : }; tail = [{ : }, { : }]; newList = [head, ...tail] .log(newList); const value 'A' const value 'B' value 'C' // -------------------- 👇 spread operator const console // expected output: // [ { value: 'A' }, { value: 'B' }, { value: 'C' } ] End In this post, we looked at the spread operator in order to convert a list into a CSV. We learned how the spread operator is the opposite of the rest parameters and how they can be used together. Then, we concluded by using the spread operator to append elements at the front of a list. Also Published On: https://micogongob.com/how-to-use-the-spread-operator-in-javascript

Using the Spread Operator in JavaScript

With the introduction of lockdown measures and brick-and-mortar stores closing their doors, retailers’ reliance on has intensified. The push to online and the need to attract and retain customers has made retail brands invest more in digital advertising and bring livestream shopping to their marketing mixes. ecommerce solutions A hot trend, livestream shopping makes online shopping more engaging and experiential while taking conversions way up. In this article, we’ll examine the phenomenon of livestream shopping, evaluate its impact on the retail sector, and zoom in on out-of-the-box and custom live video shopping tools. What is livestream shopping? The concept of live shopping goes back to the 80s and builds on the idea of video shopping. You are likely to remember QVC’s Lisa Robertson casually chatting with the viewers while demonstrating jewelry or HSN's Bobby Ray Carter showcasing fashion pieces with the viewers tuning in, seeing what’s for sale, and calling in to make a purchase. Livestream shopping — using video streaming tools to demonstrate products, communicate with an audience, and sell, all online — is a natural offshoot of this original TV-oriented model. Combining the best of ecommerce, video streaming, and social media, live commerce lets people shop anytime and from anywhere, encourages real-time communication, and sparks interactivity, appealing to today’s tech-savvy, sit-at-home customersLivestream shopping can take different forms, from live events and auctions to online marketplaces and influencer streams. Irrespective of the format, viewers can tune into a live shopping session, leave comments, ask questions, and communicate with a host or with one another. And once they are ready to purchase, they can click a link embedded in a shoppable video and proceed to checkout. Live shopping provides for personalization, too. Although one-to-many live streams, when products are marketed to mass online audiences, are more popular, brands can pay special treatment to their repeat buyers through one-to-one livestream sessions. With private dialogues between a host and a customer, brands can produce higher conversion rates and average order values and increase the customers’ chance of returning. How does livestream shopping work? Suppose you want to tap into livestream shopping. In that case, you have three options: 1. Leverage social media’s streaming functionality conditions and predicting patient outcomes 2. Market your products on an existing video commerce platform or a marketplace with streaming features 3. Opt for a custom livestream shopping solution 1. Social media The most popular and perhaps most straightforward way to start with livestream shopping is going live on popular social media apps. Luckily, most social media platforms have introduced livestream and ecommerce functionality in response to the video shopping trend. Instagram, for instance, has added built-in checkout capabilities throughout its apps, including Instagram Live, which, by the way, has seen a 70% increase in the number of viewers in 2020. Facebook, too, is developing native livestream shopping tools. In turn, TikTok is partnering with ecommerce giant Walmart for the latter to market their products online. The critical advantage of turning to social media is the possibility to team up with influencers, who play a crucial role in the success of livestream shopping, right where their audiences are. A study from Adweek reports that 49% of respondents actively follow social media influencers and based upon an influencer’s recommendation. So, having influencers as hosts may help in developing trust and loyalty among customers. However, as social media platforms generate income via media buy, they focus on being social. So, ecommerce functionality is still limited to select brands. As a result, shoppers have to be routed to other platforms or websites to finalize purchases, which disrupts the whole retailtainment experience. have made a purchase 2. Out-of-the-box live shopping apps Another way to start capitalizing on the video shopping trend involves joining an existing livestream shopping platform or an ecommerce marketplace with livestreaming features. And these span from video commerce giants like Amazon Live and Taobao, live shopping startups like Bambuser and Spin, and niche live commerce apps like NTWRK. The essential advantage is that most out-of-the-box video shopping apps feature built-in integrations with popular ecommerce platforms and payment gateways as compared to the previous option. So, users don’t have to be routed to other sites to complete a purchase and can enjoy an uninterrupted video shopping experience. The fundamental limitation is that some livestream shopping platforms, especially larger ones and those with established audiences, expect a brand to meet a range of requirements, like having a steady business history, and charge commissions. Amazon Live, for instance, charges up to 10% on each purchase. Another downside is rising competition — with millions of livestream views, it can be tricky for retailers to stand out. 3. Custom live video shopping applications Retailers who seek to make video shopping an essential part of their marketing kit may consider developing a custom retail solution with livestream functionality. Going custom is undoubtedly more effort-intensive than the previous two options. But a custom app can let retailers take maximum advantage of the video shopping trend with tailored functionality, high-quality streaming, advanced analytics, and maybe even some extra features, like AI-powered video content personalization. If you do opt for a we recommend starting with an MVP. However, it is essential to plan for the future and make sure your app will handle the growing number of shoppers. For that, make time to choose a reliable content delivery network (CDN). Looking for the perfect CDN, make sure it has a robust server infrastructure close to your target locations and pay attention to the bandwidth. With the basics in place, you can think about MVP features. The essentials to include in the first app version are: custom livestream shopping app, User registration and authorization Online streaming as such Live chatting Checkout and payment Server-side stream processing With the subsequent iterations, you can add everything from different signup options, personal customer profiles, search options, the possibility to save streams as on-demand videos, advanced analytics, streaming content personalization, and more What’s the livestream shopping market size? The current livestream shopping trend has its roots in China. China’s ecommerce giants started holding live shopping events hosted by local celebrities back in the 2010s. The US has begun to catch up with the trend, too. The US livestreaming market is expected to by the end of 2021 and hit $25 billion by 2023, according to Coresight Research. The market value is still incomparable to that in China, where it comprised $150 billion in 2020 and is expected to reach $300 billion in 2021. The companies that look prominent on the US livestream shopping market today include: reach $11 billion Launched in 2018, the platform has a 2-million follower base, which is constantly growing. According to Bloomberg, since its premiere, TalkShopLive’s sales have increased sevenfold TalkShopLive. . Having started as a directory for American retailers to reach Chinese travelers, the platform has turned into a profitable livestream shopping endeavor that has recently expanded into and is gaining popularity on the American market ShopShops . The mobile app uses live shows to sell limited edition products. The company has seen its sales double in 2020, with some shows reaching $1 million in sales in approximately 10 minutes NTWRK The ecommerce giant has rolled out its video streaming service with lifestyle, fitness, and cooking streams Amazon. The company has introduced Shoploop — a video shopping platform that features 90-second shoppable stories Google. Its Live Shopping feature allows creators and brands to sell products during Instagram Live broadcasts. The functionality is available to business accounts Instagram. An explosion of digital media and video content consumption, an increasing impact celebrities and influencers have on shopping decisions, and the push to online boosted by the COVID-19 pandemic allow projecting that more US retail brands will move on the live shopping trend in the near future. Companies profiting from livestream shopping today Nordstrom Nordstrom, an American department store chain, launched its own in 2020. Following the launch, the brand hosted over 50 virtual events, translating the store experience into the digital world, and plans on carrying out their livestream shopping endeavors through 2021. During each event, customers can shop the fashion products mentioned in the video and participate in a live chat. The company’s efforts are paying off. In 2020, 54% of the brand’s sales came from digital channels, while virtual styling accounted for 30% of all styling appointments. shoppable livestream channel CAIA Cosmetics, Lyko CAIA Cosmetics and Lyko, Swedish cosmetic and hair care brands, have partnered with Bambuser, a video commerce startup, to power their marketing with live shopping. CAIA Cosmetics, for instance, holds make-up tutorials with the brand’s founder Bianca Ingrosso and claims to be very satisfied with the customers’ response. The , with 17% of the viewers participating in the live chat and 60% — engaging through likes. The first live stream ended up with a 5% conversion rate, which is higher than the brand’s website conversions. Lyko, in turn, chose to kick off their live broadcasts by offering tips and gifts during the holiday season. The brand had an average viewing time of 16.5 minutes, 17% chatting and 29% liking engagement, and a 22% conversion rate. average stream viewing time comprises 11 minutes Quivr Quivr, a coffee and tea manufacturer, has tried social media to market their products via video streams but eventually landed on Amazon Live as their video commerce platform. Quivr’s co-founder Ash Crawford says that the of about 150% the next 24 hours after a live stream. brand sees a guaranteed increase in sales Livestream shopping trends and predictions The possibilities for further evolution and expansion of livestream shopping are numerous. Introducing AI, for example, can help retailers create highly personalized channels of live, interactive content created or curated specifically for a given customer. and VR, in turn, would be of help in creating immersive video shopping experiences for sit-at-home customers, for example, transporting them to the front row of a fashion show with a host ready to answer questions. Leveraging AR In the shorter perspective, brands will continue to extend their reach to more customers through live streams, analysts say. Considering the drastic change in shopping habits and the push to online spaces that are expected to last, livestream shopping is likely to extend in the post-pandemic world. Livestream shopping offers an opportunity to bring the best of physical shopping to online customers. If you’re ready to seize the opportunity, contact the ITRex team!

An Introduction to Livestream Shopping

You have some new innovative, fun idea of social media.You want a secure, smart, fast and scalable app.This is how you can achieve this on limited time and budget. This article based on personal experience and hours of research. It will mostly contain code examples. — scalable servers and extra tools — real time database — security and performance — emails Ingredients: Node.js GCP Firebase Cloudflare Mailgun We will use one App engine and split it into 3 microservices1st microservice — Backend server2nd microservice — Cron server3rd microservice — Web server I will use for the client side.The Backend solutions are good for other client options. React Before you write the first line of code You need to set your goals for the first release.It’s important to decide what is necessary for the first version and what you can add after. Choose a name for your social media app.Check if the domain is available. You can use Google domains or GoDaddy for this. Open accounts and create new project at GCP, Firebase Cloudflare and Mailgun. Open developers account on Facebook and Instagram to enable login options. (Make sure you are sending a review request, it might take some time). Backend API server Goals: Client side friendlySecurityAuto scalingEasy to maintainKeep logsReal time reports on bugs Yes, you can achieve those goals with a traditional RESTful API server, but, I would like to show you another alternative which will save you a lot of time. We are going to use , Firebase real-time database and Google App engine to create that. Firebase queue This will be my folders hierarchy. 📁config📁keys➥serviceAccountKey.jsonconfiguration.jsonindex.js 📁services📁firebase➥index.js📁image_manipulation➥index.js📁notifications➥index.js 📁utils➥index.js app.jsapp.yamlpackage.json Node dependencies: "fcm-node": "^1.0.15","firebase": "^5.4.0","firebase-admin": "^6.0.0","firebase-queue": "^1.6.1","mailgun-js": "^0.20.0" Image manipulation dependencies: "@google-cloud/storage": "^1.5.1","@google-cloud/vision": "^0.14.0","image-size": "^0.6.2","imagemin": "^5.3.1","imagemin-giflossy": "^5.1.10","imagemin-jpeg-recompress": "^5.1.0","sharp": "^0.18.4","smartcrop": "^1.1.1","smartcrop-sharp": "^1.0.5" Let's start with the config/configuration.json: {"firebase": {"projectId": "YOUT_PROJECT_ID","url": "YOUT_PROJECT_DB_URL","serviceAccount": "config/keys/serviceAccountKey.json"},"mailgun": {"apiKey": "YOUT_MAILGUN_KEY","domain": "YOUT_MAILGUN_DOMAIN"},"app": { config/index.js (note that there is configuration.dev.json) config = process.env.NODE_ENV === 'production'? require('config/configuration.json'): require('config/configuration.dev.json'); const module.exports = config; app.yaml : api : nodejs : flex : : prod service runtime env env_variables NODE_ENV services/firebase/index.js admin = require('firebase-admin'); config  = require('config'); const const admin.initializeApp({credential: admin.credential.cert(require(config.firebase.serviceAccount)),databaseURL: config.firebase.url}); db = admin.firestore(); tasksDB = admin.database(); tasksQueue = tasksDB.ref('tasks_queue/'); const const const db.settings({ timestampsInSnapshots: }); true // Here we will write our db functions module.exports = {tasksQueue Now we can start writing our server ( app.js) Personally, I like to set my require() paths like this: (you can read more about it ) here process.env.NODE_PATH = __dirname;require('module'). ._initPaths(); Module We will start with the setup and add three simple ‘tasks’ and . saveUser , updateUser likePost function will save the incoming task in our log table at Firebase. addLogs will email the task data with the error message in case of an error. mailError And will listen to the table on Firebases real time DB and whenever a new task written in that table it will process the task according to its type. If there are more tasks than he can handle App engine will autoscale. processTask tasks_queue As you can see we need to add some function to our Firebase endpoint file setData(ref, data){ ( ref === 'string') db.doc(ref).set(data); ref.set(data);} function if typeof return return updateData(ref, data){ ( ref === 'string') db.doc(ref).update(data); ref.update(data) function if typeof return return ateLogs(date, taskType, time, task) {tasksDB.ref(`logs/${date}/${taskType}/${time}/task`).set(task) likePost(postId){ Promise((resolve, reject) => { postRef = db.collection('posts').doc(postId); db.runTransaction((transaction) => { transaction.get(postRef).then((postDoc) => { (!postDoc.exists) resolve(); newLikes = postDoc.data().likes + 1;transaction.update(postRef, { likes: newLikes });});}).then(resolve).catch(reject);});} ion return new const return return if const // Don't forget to add them to the module exports This is how the tasks table will look like: Another thing we may want to add to our server is image manipulation. Let's say, a user uploaded a photo to our fun social media, We will need to compress that image, Maybe alert us if it is offensive and we may even want to create a smart thumbnail for it. We will add the task to handle that request Now, lets have a look at services/image_manipulation/index.js (I will not get into details on this one) Notifications Our most valuable retention tool. We need to inform the users if, for example, someone like their post. We want to be able to send messages and notifications to users across platforms, Android, iOS, and the web. A good thing will be also to save it in our DB so we can display all of the notification inside the app. Let’s change our like_post function likePost(data = {}) { Promise((resolve, reject) => { { user, postId, postOwner, } = data;Promise.all([db.likePost(postId),db.newNotification('like', user, postOwner),]).then(resolve).catch(reject);});} function return new const And add to our Firebase endpoint file newNotification fcm = require('services/notifications'); notificationsText = require('texts/en/notifications'); const const newNotification(tag, from, to) { Promise((resolve, reject) => { notification = notificationsText(tag, from, to); (to.deviceToken) fcm.sendNativeNotification(notification, deviceToken); newNotificationRef = db.collection(`messages/${to.uid}/notification`).doc();newNotificationRef.set(notification).then(resolve).catch(reject);});} function return new const if const services/notifications/index.js So there you have it, API server that does exactly what you need to bootstrap (Well, you will still need to write functions that will handle your client tasks). rules can be easily set through the rules tab on the console. You can read more about it . Security Firebase here Cron server If we need some schedule functions to run in the background. This is how we will achieve that: In this example our schedule function will like a random post The folder hierarchy 📁bin➥www 📁config📁keys➥serviceAccountKey.jsonconfiguration.jsonindex. 📁cron_jobs➥ index.jsrandom_like.js 📁services📁firebase➥index.js 📁utils➥index.js app.jsapp.yamlcron.yamlpackage.json Node dependencies: "express": "^4.16.3","firebase": "^5.4.0","firebase-admin": "^6.0.0" The configurations remain the same as the API server. app.yaml : cron : nodejs : flex : : production service runtime env env_variables NODE_ENV cron.yaml :- : Like a Random Post : /like_random_post : every 10 minutes : Etc/GMT : cron cron description url schedule timezone target app.js process.env.NODE_PATH = __dirname;require('module'). ._initPaths(); Module express = require('express'); app = express();app.enable('trust proxy'); const const app.use('/', require('cron_jobs')); app.use((err, req, res, next) => res.status(500).send(err.message || 'Something broke!')); module.exports = app; cron_jobs/inde express = require('express'); randomLike = require('cron_jobs/random_like'); router = express.Router(); t const const // [START routing]router.get('/like_random_post', randomLike);// [END routing] module.exports = router; cron_jobs/random_like.js At the Firebase endpoint file // addTask(type, data){ tasksDB.ref('tasks_queue/tasks').push({ type, data });} function return As you can see at the cron.yaml file, a GET request will be fired every 10 minutes to the /like_random_post endpoint (target: cron microservice). The function gets a random post from the DB and add a ‘like_post’ task to our tasks queue. That’s conclude the first part of our story, by now you should have a working server that can be use for any type of app, native or web. Part 2 — Frontend Web server This server will serve pages to the users. Basically, we will need only one route since we will handle all the routing with React-router. However, when our users will share their posts to others social media, we will want it to be attractive. Rich previews are a great tool to achieve that. WebApp In order to support Rich preview we will need to adjust our html file for every post. I will add an example code here of how it can be done easily, this is just an example there are others ways to do it. Folder hierarchy: 📁bin➥www 📁config📁keys➥serviceAccountKey.jsonconfiguration.jsonindex. 📁routes➥ index.jspost.js 📁services📁firebase➥index.js 📁utils➥index.js 📁views➥index.js➥html_template.js app.jsapp.yamlpackage.json app.yaml : nodejs : flex : : production runtime env env_variables NODE_ENV app.js As you can see, we will server our files zipped for better performance. routes/index.js 'use strict'; express = require('express'); router = express.Router(); htmlTemplate = require('../views'); const const const // [START post]router.get('/post/*', require('./post'));// [END post] // [START app]router.get('/*', (req, res) => {// console.log(req); res.send(htmlTemplate())});// [END app] return module.exports = router; routes/post.js html_template.js Web App There are plenty of great ‘how to build a React web app’ articles out there. Here I will focus on some social media web app challenges. Goals: FeedLive ListenersImages uploadsRich previews Let’s start with the root of our WebApp pushNotification is our the way to communicate with the users of the app, which looks something like this: For that we will use ‘toastr js’. toastr 'toastr'; import from pushNotification(message, type = 'error') {toastr.options.positionClass = 'toast-bottom-full-width';toastr.options.showMethod = 'slideDown';toastr.options.hideMethod = 'slideUp';toastr.options.hideDuration = 300;toastr.options.newestOnTop = ;toastr.remove(); (type) { 'info':toastr.info(message); ; 'error':toastr.error(message); ; 'success':toastr.success(message); ; 'warning':toastr.warning(message); ; :toastr.error(message); ;}} export default function false switch case break case break case break case break default break Firebase end-point file: import firebase from 'firebase/app';import 'firebase/auth';import 'firebase/storage';import 'firebase/firestore';import 'firebase/database';import config from 'config'; class Firebase { static serverTimestamp() {return firebase.firestore.FieldValue.serverTimestamp();} constructor() {firebase.initializeApp({apiKey: config.firebase.api_key,authDomain: config.firebase.domain,projectId: config.firebase.project_id,storageBucket: config.firebase.bucket,});this.auth = firebase.auth();this.db = firebase.firestore();this.db.settings({ timestampsInSnapshots: true }); .tasksDB = firebase.database();this.tasks = .tasksDB.ref('tasks_queue/tasks');this.storage = firebase.storage();} this this } export default Firebase; Feed 📁feed📁view➥feed_item.js➥index.js➥index.js Hopefully we will need to handle a lot of posts. In order to manage it correctly, we need: 1. Paginate data with query cursors 2. Highly efficient infinite scrollable container 3. On-view post updates listener “Controller”: For infinite scrollable container we will use ‘react-infinite’ “View”: For the post item we will want to add some listeners. For example, likes — counter should be constantly updated in real time. However, setting listeners to the huge amount of posts can hit our webapp performance significantly, in order to make it efficient we will listen to changes only the posts that are in-view and use ‘react-visibility-sensor’ to do that. Let’s add some functions to our Firebase end-point file getRecentPosts(startAfter = ) { postRef = startAfter? .db.collection('posts').orderBy('created_at', 'desc').startAfter(startAfter.created_at).limit(config.app.feed_paging): .db.collection('posts').orderBy('created_at', 'desc').limit(config.app.feed_paging); postRef.get().then(snapshot => snapshot.docs.map(doc => doc.data()));} null const this this return postChangesListener(postId, callback) { .db.collection('posts').doc(postId).onSnapshot((doc) => { (callback) callback(doc.data());});} return this if addTask(type, data) {return this.tasks.push({ type, ...data });} Image upload We want our users to be able to upload photos to our web app, we want to create thumbnails for those photos, we don’t want them to wait a long time for it to finish. In order to accomplish that we will need to lower the photo size (for faster upload and to create the thumbnail). UploadPhoto Component: When we resize a photo we need to take its exif data into account, otherwise we may encounter orientation issues. ImageTools: Login example Final Firebase end-point file Note that the Firestore data manipulation doesn’t happen on the client side, but instead we add tasks to our server through Firebase RD. This is only for security reasons. I am using webpack 4 and this is my webpack.config.js, if you need code splitting and gzip you can use it as an example. After you finish writing your first version and deploy everything to GCP you can now add your App Engine records to your Cloudflare DNS settings and get a free performance and security boost including SSL, CDN and much more. Here’s my final tip for building your new fun social media: for everything. That is how you would know what your users are doing. It will help you improve and invest your time on the things that matter to them. include analytics I would love to get your comments and suggestions. If you have any questions, feel free to contact me at my . Thank you for reading! Spread the love :) linkedin

How to build a fun social media on limited time and budget

NOTE: I have originally published this article a while ago on . Toptal blog I once drove an Audi with a V8 twin-turbo engine inside, and its performance was incredible. I was driving at around 140MPH on IL-80 highway near Chicago at 3AM when there was nobody on the road. Ever since then, the term “V8” has become associated with high performance to me. is a platform built on Chrome’s V8 engine for easy building of fast and scalable network applications. Node.js JavaScript Although Audi’s V8 is very powerful, you are still limited with the capacity of your gas tank. The same goes for Google’s V8 — the JavaScript engine behind Node.js. Its performance is incredible and there are many reasons why Node.js , but you’re always limited by the heap size. When you need to process more requests in your Node.js application you have two choices: either scale vertically or scale horizontally. Horizontal scaling means you have to run more concurrent application instances. When done right, you end up being able to serve more requests. Vertical scaling means that you have to improve your application’s memory usage and performance or increase resources available for your application instance. works well for many use cases Debugging Memory Leaks in Node.js Applications Recently I was asked to work on a Node.js application for one of my Toptal clients to fix a memory leak issue. The application, an API server, was intended to be able to process hundreds of thousands of requests every minute. The original application occupied almost 600MB of RAM and therefore we decided to take the hot API endpoints and reimplement them. Overhead becomes very pricey when you need to serve many requests. For the new API we chose restify with native MongoDB driver and Kue for background jobs. Sounds like a very lightweight stack, right? Not quite. During peak load a new application instance could consume up to 270MB of RAM. Therefore my dream of having two application instances per 1X Heroku Dyno vanished. Node.js Memory Leak Debugging Arsenal Memwatch If you search for “how to find leak in node” the first tool you’d probably find is . The original package was abandoned a long time ago and is no longer maintained. However you can easily find newer versions of it in GitHub’s . This module is useful because it can emit leak events if it sees the heap grow over 5 consecutive garbage collections. memwatch fork list for the repository Heapdump Great tool which allows to take heap snapshot and inspect them later with Chrome Developer Tools. Node.js developers Node-inspector Even a more useful alternative to heapdump, because it allows you to connect to a running application, take heap dump and even debug and recompile it on the fly. Taking “node-inspector” for a Spin Unfortunately, you will not be able to connect to production applications that are running on Heroku, because it does not allow signals to be sent to running processes. However, Heroku is not the only hosting platform. To experience node-inspector in action, we will write a simple Node.js application using restify and put a little source of memory leak within it. All experiments here are made with Node.js v0.12.7, which has been compiled against V8 v3.28.71.19. var restify = require('restify');var server = restify.createServer();var tasks = [];server.pre(function(req, res, next) {  tasks.push(function() {    return req.headers;  });  // Synchronously get user from session, maybe jwt token  req.user = {    id: 1,    username: 'Leaky Master',  };  return next();});server.get('/', function(req, res, next) {  res.send('Hi ' + req.user.username);  return next();});server.listen(3000, function() {  console.log('%s listening at %s', server.name, server.url);}); The application here is very simple and has a very obvious leak. The array would grow over application lifetime causing it to slow down and eventually crash. The problem is that we are not only leaking closure but entire request objects as well. tasks GC in V8 employs stop-the-world strategy, therefore it means more objects you have in memory the longer it will take to collect garbage. On log below you can clearly see that in the beginning of the application life it would take an average of 20ms to collect the garbage, but few hundred thousand requests later it takes around 230ms. People who are trying to access our application would have to wait longer now because of GC. Also you can see that GC is invoked every few seconds which means that every few seconds users would experience problems accessing our application. And delay will grow up until application crashes. 230ms [28093] 7644 ms: Mark-sweep 10.9 (48.5) -> 10.9 (48.5) MB, 25.0 ms [HeapObjectsMap::UpdateHeapObjectsMap] [GC in old space requested]. [28093] 7717 ms: Mark-sweep 10.9 (48.5) -> 10.9 (48.5) MB, 18.0 ms [HeapObjectsMap::UpdateHeapObjectsMap] [GC in old space requested]. [28093] 7866 ms: Mark-sweep 11.0 (48.5) -> 10.9 (48.5) MB, 23.2 ms [HeapObjectsMap::UpdateHeapObjectsMap] [GC in old space requested]. [28093] 8001 ms: Mark-sweep 11.0 (48.5) -> 10.9 (48.5) MB, 18.4 ms [HeapObjectsMap::UpdateHeapObjectsMap] [GC in old space requested]. ... [28093] 633891 ms: Mark-sweep 235.7 (290.5) -> 235.7 (290.5) MB, 357.3 ms [HeapObjectsMap::UpdateHeapObjectsMap] [GC in old space requested]. [28093] 635672 ms: Mark-sweep 235.7 (290.5) -> 235.7 (290.5) MB, 331.5 ms [HeapObjectsMap::UpdateHeapObjectsMap] [GC in old space requested]. [28093] 637508 ms: Mark-sweep 235.7 (290.5) -> 235.7 (290.5) MB, 357.2 ms [HeapObjectsMap::UpdateHeapObjectsMap] [GC in old space requested]. These log lines are printed when a Node.js application is started with the flag: –trace_gc node --trace_gc app.js Let us assume that we have already started our Node.js application with this flag. Before connecting the application with node-inspector, we need to send it the SIGUSR1 signal to the running process. If you run Node.js in cluster, make sure you connect to one of the slave processes. kill -SIGUSR1 $pid # Replace $pid with the actual process ID By doing this, we are making the Node.js application (V8 to be precise) enter debugging mode. In this mode, the application automatically opens the port 5858 with . V8 Debugging Protocol Our next step is to run node-inspector which will connect to the debugging interface of the running application and open another web interface on port 8080. $ node-inspector Node Inspector v0.12.2 Visit http://127.0.0.1:8080/?ws=127.0.0.1:8080&port=5858 to start debugging. In case the application is running on production and you have a firewall in place, we can tunnel remote port 8080 to localhost: ssh -L 8080:localhost:8080 admin@example.com Now you could open your Chrome web browser and get full access to Chrome Development Tools attached to your remote production application. Unfortunately, Chrome Developer Tools will not work in other browsers. Let’s Find a Leak! Memory leaks in V8 are not real memory leaks as we know them from C/C++ applications. In JavaScript variables do not disappear into the void, they just get “forgotten”. Our goal is to find these forgotten variables and remind them that Dobby is free. Inside Chrome Developer Tools we have access to multiple profilers. We are particularly interested in which runs and takes multiple heap snapshots over time. This gives us a clear peek into which objects are leaking. Record Heap Allocations Start recording heap allocations and let’s simulate 50 concurrent users on our home page using Apache Benchmark. ab -c 50 -n 1000000 -k http://example.com/ Before taking new snapshots, V8 would perform mark-sweep garbage collection, so we definitely know that there is no old garbage in the snapshot. Fixing the Leak on the Fly After collecting heap allocation snapshots over a period of we end up with something like the following: 3 minutes We can clearly see that there are some gigantic arrays, a lot of IncomingMessage, ReadableState, ServerResponse and Domain objects as well in heap. Let’s try to analyze the source of the leak. Upon selecting heap diff on chart from 20s to 40s, we will only see objects which were added after 20s from when you started the profiler. This way you could exclude all normal data. Keeping note of how many objects of each type are in the system, we expand the filter from 20s to 1min. We can see that the arrays, already quite gigantic, keeps growing. Under “(array)” we can see that there are a lot of objects “(object properties)” with equal distance. Those objects are the source of our memory leak. Also we can see that “(closure)” objects grow rapidly as well. It might be handy to look at the strings as well. Under the strings list there are a lot of “Hi Leaky Master” phrases. Those might give us some clue too. In our case we know that the string ”Hi Leaky Master” could only be assembled under the “GET /” route. If you open retainers path you will see this string is somehow referenced via , then there is context created and all this added to some giant array of closures. req So at this point we know that we have some kind of gigantic array of closures. Let’s actually go and give a name to all our closures at real-time under sources tab. After we are done editing the code, we can hit CTRL+S to save and recompile code on the fly! Now let’s record another and see which closures are occupying the memory. Heap Allocations Snapshot It’s clear that is our villain. Now we can see that closures are being added to some array named in the global space. SomeKindOfClojure() SomeKindOfClojure() tasks It’s easy to see that this array is just useless. We can comment it out. But how do we free memory the memory already occupied? Very easy, we just assign an empty array to and with the next request it will be overridden and memory will be freed after next GC event. tasks Dobby is free! Life of Garbage in V8 Well, V8 JS does not have memory leaks, only forgotten variables. V8 heap is divided into several different spaces: : This space is relatively small and has a size of between 1MB and 8MB. Most of the objects are allocated here. New Space : Has objects which may have pointers to other objects. If object survives long enough in New Space it gets promoted to Old Pointer Space. Old Pointer Space : Contains only raw data like strings, boxed numbers and arrays of unboxed doubles. Objects that have survived GC in the New Space for long enough are moved here as well. Old Data Space : Objects which are too big to fit in other spaces are created in this space. Each object has it’s own ‘ed region in memory Large Object Space mmap : Contains assembly code generated by the JIT compiler. Code space : This space contains s, s, and s. This is used to simplify garbage collection. Cell space, property cell space, map space Cell PropertyCell Map Each space is composed of pages. A page is a region of memory allocated from the operating system with mmap. Each page is always 1MB in size except for pages in large object space. V8 has two built in garbage collection mechanisms: Scavenge, Mark-Sweep and Mark-Compact. Scavenge is a very fast garbage collection technique and operates with objects in . Scavenge is the implementation of . The idea is very simple, is divided in two equal semi-spaces: To-Space and From-Space. Scavenge GC occurs when To-Space is full. It simply swaps To and From spaces and copy all live objects to To-Space or promote them to one of the old spaces if they survived two scavenges, and is then entirely erased from the space. Scavenges are very fast however they have the overhead of keeping double sized heap and constantly copying objects in memory. The reason to use scavenges is because most objects die young. New Space Cheney’s Algorithm New Space Mark-Sweep & Mark-Compact is another type of garbage collector used in V8. The other name is full garbage collector. It marks all live nodes, then sweeps all dead nodes and defragments memory. GC Performance and Debugging Tips While for web applications high performance might not be such a big problem, you will still want to avoid leaks at all costs. During the mark phase in full GC the application is actually paused until garbage collection is completed. This means the more objects you have in the heap, the longer it will take to perform GC and the longer users will have to wait. Always give names to closures and functions It’s much easier to inspect stack traces and heaps when all your closures and functions have names. db.query('GIVE THEM ALL', function GiveThemAllAName(error, data) {    ...}) Avoid large objects in hot functions Ideally you want to avoid large objects inside of hot functions so that all data is fit into . All CPU and memory bound operations should be executed in background. Also avoid deoptimization triggers for hot functions, optimized hot function uses less memory than non-optimized ones. New Space Hot functions should be optimized Hot functions that run faster but also consume less memory cause GC to run less often. V8 provides some helpful debugging tools to spot non-optimized functions or deoptimized functions. Avoid polymorphism for IC’s in hot functions Inline Caches (IC) are used to speed up execution of some chunks of code, either by caching object property access or some simple function. obj.key function x(a, b) {  return a + b;}x(1, 2); // monomorphicx(1, “string”); // polymorphic, level 2x(3.14, 1); // polymorphic, level 3 When is run for the first time, V8 creates a monomorphic IC. When you call a second time, V8 erases the old IC and creates a new polymorphic IC which supports both types of operands integer and string. When you call IC the third time, V8 repeats the same procedure and creates another polymorphic IC of level 3. x(a,b) x However, there is a limitation. After IC level reaches 5 (could be changed with flag) the function becomes megamorphic and is no longer considered optimizable. –max_inlining_levels It’s intuitively understandable that monomorphic functions run the fastest and also have a smaller memory footprint. Don’t add large files to memory This one is obvious and well known. If you have large files to process, for example a large CSV file, read it line-by-line and process in little chunks instead of loading the entire file to memory. There are rather rare cases where a single line of csv would be larger than 1mb, thus allowing you to fit it in . New Space Do not block main server thread If you have some hot API which takes some time to process, such as an API to resize images, move it to a separate thread or turn it into a background job. CPU intensive operations would block main thread forcing all other customers to wait and keep sending requests. Unprocessed request data would stack in memory, thus forcing full GC to take longer time to finish. Do not create unnecessary data I once had a weird experience with restify. If you send a few hundred thousand requests to an invalid URL then the application memory would rapidly grow on up to hundred megabytes until a full GC kicks in a few seconds later, which is when everything would go back to normal. Turns out that for each invalid URL, restify generates a new error object which includes long stack traces. This forced newly created objects to be allocated in rather than in . Large Object Space New Space Having access to such data could be very helpful during development, but obviously not required on production. Therefore the rule is simple — do not generate data unless you certainly need it. Last, but certainly not the least, is to know your tools. There are various debuggers, leak cathers, and usage graphs generators. All those tools can help you make your software faster and more efficient. Conclusion Understanding how V8’s garbage collection and code optimizer works is a key to application performance. V8 compiles JavaScript to native assembly and in some cases well written code could achieve performance comparable with GCC compiled applications. And in case you are wondering, the new API application for my Toptal client, although there is room for improvement, is working very well! Joyent recently released a new version of Node.js which uses one of the latest versions of V8. Some applications written for Node.js v0.12.x may not be compatible with the new v4.x release. However, applications will experience tremendous performance and memory usage improvement within the new version of Node.js. Originally published at www.toptal.com .

hDebugging Memory Leaks in Node.js Applications

Selling indie games — part 1 Part 1 in a series on selling indie games, primarily based on my own experiences, supplemented with those of some other studios. This a story about the sales of indie games and written articles on game websites. It’s goal is to provide you with some insights and some examples. Little disclaimer: these are OUR experiences, yours may vary wildly and are not mutually exclusive. I personally have the utmost respect for all websites mentioned and do consider them important, no matter their impact on our sales. Their primary goal is not selling our game, but informing their readers. However, it’s interesting to share the (immediate) effects we were able to measure. Two months after the launch of our latest game, , we decided to create a new trailer. There really wasn’t any good reason for doing so, except me wanting to see if I could cut a game trailer to a waltz I’d written before. Unexplored We were pleased with the result and uploaded the trailer to our Steam-page, and that was it. Which was stupid, considering the fact that now only people already interested in the game would see it. It should lure in potential new players. So, we drafted a press release. Which was hard, because, again, there was no real reason for it other than . However, our lead developer had recently made some changes to the UI (well, he hid it), and we decided to go with that. Please, put this fancy trailer on your website The header was a bit of an exaggeration, to say the least, but it provided the excuse we needed to send the new trailer to 1,200 game websites. Unexplored kills UI after player feedback After that, we just had to wait to see if someone would pick it up. The trailer would probably end up on some small outlet with barely enough visitors to generate some views, let alone some actual sales. Or, you know, on PC Gamer. The had some fun words… article …but we were very pleased with it for even more reasons. To understand why, you have to be aware of the decision we made months earlier. Unexplored’s graphical style is functional but doesn’t appeal to everyone, the gameplay is silky smooth but not particularly groundbreaking, and being a roguelite, a crowded genre, we really needed something to make the game stand out. We were in luck. Our lead is somewhat of an in procedural content generation and he had cooked up a brilliant little idea to drastically improve the quality of generated dungeons. developer expert You see, usually random generators use branching paths to create dungeons: At its core, Unexplored uses an actual innovative, different idea, cycles: By making the computer ‘think’ in cycles instead of branching paths, it actually mimics a human level designer. The results are great, levels feel much more handmade, compared to existing generated dungeons. We decided to make this our core marketing message and even came up with a new term: Cyclic Dungeon Generation. Back to the PC Gamer article a couple of months later. It more than just touched upon our generator: I had our lead developer explain the whole theory in , in front of a camera, as part of our marketing approach. The PC Gamer article featured that video as well. less than a minute So, our brand new trailer, some nice words, and the whole Cyclic Dungeon Generation-story were put in front of potentially millions of readers. I mean, just look at those numbers: Surely, this was it! Our game would go mainstream, and I had a pretty sweet rev share deal… Here’s our sales chart, the arrow indicating the date the article was published: Enter: deception time. That spike is barely a hill, a far cry from the Mount Everest I was eagerly expecting. Put in actual numbers, this is the estimated increase in total sales, due to that wonderful PC Gamer-article: To us, the less than stellar sales drive from the PC Gamer-article came as a surprise, especially since we had a polar opposite experience just a few weeks earlier. I’ll explain. First, a bit of background about the game: Most of Unexplored (design, art, code) is made by one man, our PCG-expert and Lead Developer, Joris Dormans. We had a writer, a couple of guys doing additional coding and me (music, audio, marketing). We launched Unexplored as an Early Access game on Steam (which I believe is still perfect for non-linear, non-narrative games). It did sell, but not tremendously, although the player feedback was something else entirely (and still is, considering the current on our Steam-page). 96% positive rating In the weeks prior to the actual launch I pushed out more press releases and personal e-mails than I care to remember. All of them focusing on our core marketing message in a quote-friendly fashion, featuring a gameplay trailer, screenshots, Steam-keys, the likes. That resulted in some coverage, but nothing great. And that’s exactly how our launch day went down. Considering the fact that your Day One sales are the most you’ll ever get, this was disappointing. Based on these 243 units, we were expecting to sell another ~100 units the next day and maybe reach 500 in the first week. Since we were selling for $10, even less during launch week, the game was on life-support. Plans for updates and extensions were iced. We weren’t very happy. Then, this tweet: It turned out, Adam Smith is a writer for Rock Paper Shotgun, a website I was fuming about mere hours before. Fuming, because I couldn’t understand that so far they had completely ignored our launch. If there was ONE large site I expected to cover Unexplored, it was the indie-friendly PC games website . Rock Paper Shotgun And now they would. Next day, next tweet: Mr. Smith wasn’t kidding: The reviewer really GOT our game. He loved it. The ended with this line: raving review Unexplored is almost certainly going to be one of my favourite games of the year Beside the Rock Paper Shotgun-article, we didn’t receive any coverage that day, in any form. Our modest frontpage feature didn’t change, but still, this is what Day #2 looked like in sales: It’s impossible to say this with certainty, but at least to us, it looks like the RPS-article provided our sales with a much needed boost. It got Unexplored off life support and even provided the game with this little long-term bonus: RPS’s Steam curator channel is one of the largest, boosting the visibility (ie. discoverability) of Unexplored on the store front for a long time. A couple of weeks later, RPS decided to do an actual follow-up article. As part of a series called , the site dove deep into the inner workings of Unexplored’s Cyclic Dungeon Generator. The Mechanic The effects of this article were very pronounced, especially since we didn’t receive any other coverage that week. 400 units may not be a lot for even a mid-sized studio. For 1.5 people, a turnover of $4,000 in four days is really good. Then again, even before the PC Gamer-article, we had coverage which didn’t create spikes. Take Indiekings, by far the best game talkshow in the Netherlands, part of Gamekings, a popular Dutch TV-show/website/game-expo-powerhouse. The host and I talked for about indie games in general and Unexplored in particular (I’m the one with the glasses…). an hour The episodes tend to draw in 30 tot 50 thousand viewers. Based on those numbers, I was expecting to see some additional sales in our home country, The Netherlands. There were none. This chart shows Unexplored’s sales in the Netherlands for a week, the arrow indicating the publication date of the show. (Again, I’m not dismissing the program in any way, I like it, I think it’s important.) Another week, another example. The Italian edition of Eurogamer, not the smallest of names in the industry. Months after launch, they reviewed Unexplored, kind of liked it and it with a solid 8 out of 10. awarded These are our sales in Italy, that week: However, in May two articles appeared that DID create a meaningful sales spike. RPS did an on our first (free) dlc on May 5, Vice’s Waypoint published a on our post-game sequence on May 6. There’s no way to tell which article was more important, but together they drove sales significantly: article wonderful feature Trying to get my head around why some articles caused these spikes, while others didn’t do anything, I asked other developers if they were willing to share some of their data. A couple did. First, Wispfire, the people behind the amazing , an adventure game set in colonial times. They had a bit of an unfortunate launch. Due to a misunderstanding of Steam’s inner workings, they launched weeks after their original intended date (coincidentally, they ended up launching the same day as Unexplored). Herald I remember them eagerly awaiting an article that was going to appear on Polygon, a website with a target audience that seemed like a great fit for Herald. Three weeks after launch they saw their game screen-wide on the frontpage of Polygon. It was a long-read sort-of , but since it wasn’t scored, and lacked a typical review summary, I doubt many readers made it all the way to the end. Even so, the subsequent effect on Herald’ sales, was, well, surprising. review Wispfire sold 13 units on day of the article, 2 more than the day before and the day after. Maybe all of those 13 units were the result of the Polygon-article, but I’d still reckon this was Wispfire’s ‘PC Gamer-moment’. DoubleDutch, the studio behind the very successful , provided another three examples. These articles on larger websites appeared separate from each other, they were in fact the only coverage Speedrunners received in the weeks they were published. Sales spikes should be easy to spot, making the potential effect on unit sales easy to deduce. Speedrunners Or not. DoubleDutch couldn’t find any sales spikes on or after the dates of any of these articles. (Now, Speedrunners is much, much more successful than Unexplored, selling in the millions. Chances are that all of these articles actually DID contribute to unit sales to a certain degree, but that a high number of regular daily sales sort of just hid these additional sales from view.) A cautious first conclusion From what I have seen so far, it seems like the effects of written articles on sales of indie games are all over the place and thus pretty hard to predict. Of course, traditional factors like total reach and the composition of the target audiences explain a lot. But not all, far from it. That PC Gamer-article should have resulted in way more sales, if those were the defining factors. Reading all these articles however, there seems to be one element, one ingredient, which is more important than anything else: the genuine enthusiasm of the writer. In Unexplored’s case, all authors of articles that did drive sales really, really liked the game. They did not write run of the mill articles, but really tried to communicate why they thought Unexplored is a great game. Waypoint’s editor-in-chief Austin Walker tweeted about it, discussed the game in Waypoint’s podcast and streamed a couple of hours of him playing (and, thank God, beating the game whilst streaming). His article ends with this paragraph: When I first started playing , I knew immediately that it would be one of my favorites of the year. But the special way that it offers players sense of closure only makes my feelings about it even stronger. Even in a year with so many incredible games, is a gem. I just hope that it isn’t totally lost in the shuffle. Unexplored Unexplored This kind of genuine enthusiasm is contagious and, apparently, has a far greater impact on sales than sheer numbers. End of Part 1 Just recently Unexplored was featured in another PC Gamer-article, as part of a story about innovative roguelikes. Unfortunately, it is not possible to measure the impact of that particular article because it was published on the first day of a weeklong discount. However, the author of said appeared to have had a great time playing the game, and we feel it did help boost the sales — although to an unknown extent. Post scriptum: article UPDATE: A lot of people have made interesting remarks on this article on twitter. I’ll share a couple of them, without comment. — — — This article is based on parts of my talk at Devcom / Respawn in Cologne, earlier this year. In the Part 2, I’ll share some of the data I collected on the impact of streamers/Youtubers/twitchers on sales of indie games.

Stories, News & Mentions about TALKPUSH

Talkpush’s Stories on HackerNoon