UserPath

userpath

As developers, we are constantly seeking for methods to demonstrate our abilities and accomplishments. TryHackMe provides an excellent forum for cybersecurity enthusiasts to learn and demonstrate their expertise. However, manually upgrading your TryHackMe badges can be tedious. That’s why I created the TryHackMe Badge Action Workflow, a GitHub Action that automates the process of displaying your TryHackMe achievements within your GitHub profile or repository. In this blog article, I’ll explain why I created this action, how it works, and how you may use it in your own markdown files in github. Motivation: Automating the Display of Achievements TryHackMe is a tool that many people in the cybersecurity field use to sharpen their abilities. However, manually updating badges or tracking your progress might be time-consuming, especially if you want to share this information on your GitHub page. I thought of a solution that would automate this process, allowing anyone to demonstrate their TryHackMe achievements with no effort. What Is the TryHackMe Badge Action Workflow? The TryHackMe Badge Action Workflow is a custom GitHub Action that refreshes and displays your TryHackMe badge on your GitHub profile or repository README page. This action allows you to maintain your achievements up to date without having to manually download, update, or commit your badge picture each time you acquire a new one. How It Works This action updates your GitHub profile or repository README with the new image and retrieves your most recent TryHackMe badge. This is a summary of the workflow’s functions: Image Fetching: The procedure begins with obtaining your TryHackMe badge’s most recent version from a given URL.
Commit Changes: The downloaded image is then committed to your repository, guaranteeing that the most recent badge is always visible.
README Update: Lastly, the action adds the updated badge image to your README file, allowing you to view your TryHackMe accomplishments in real time. Usage Add the below script to your GitHub repository as a .yml file (e.g., update-tryhackme-badge.yml).


Configure the action with the following inputs:


GITHUB_TOKEN: Your GitHub Personal Access Token (required, set as a secret).


image_path: The path to store the downloaded badge image (defaults to ./assets/tryhackme-badge.png).


username: Your TryHackMe username (defaults to the value in a secret named THM_USERNAME).


user_id: Your TryHackMe user ID (defaults to the value in a secret named THM_USER_ID). name: Update TryHackMe Badge

on:
  schedule:
    - cron: '0 0 * * *' # Runs every day at midnight
  workflow_dispatch: # Allows manual triggering

jobs:
  update-badge:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout repository
      uses: actions/checkout@v2

    - name: Fetch TryHackMe Badge
      uses: DhanushNehru/tryhackme-badge-action-workflow@v1.0
      with:
        image_path: './assets/tryhackme-badge.png'
        username: 'your_tryhackme_username'
        user_id: 'your_tryhackme_user_id'
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} An easy-to-use yet effective solution for automating the display of your security accomplishments is the TryHackMe Badge Action Workflow. If you liked the project as a source of encouragement and support please give it a star ⭐️👇 https://github.com/marketplace/actions/tryhackme-badge-action-workflow?embedable=true I have used this in my Github README.md so feel free to check it out Thanks for reading, please give a like as a sort of encouragement and also share this post in socials to show your extended support. Connect ⬇️ X / Instagram / Github / Youtube / Newsletter / Discord As developers, we are constantly seeking for methods to demonstrate our abilities and accomplishments. TryHackMe provides an excellent forum for cybersecurity enthusiasts to learn and demonstrate their expertise. However, manually upgrading your TryHackMe badges can be tedious. That’s why I created the TryHackMe Badge Action Workflow, a GitHub Action that automates the process of displaying your TryHackMe achievements within your GitHub profile or repository. In this blog article, I’ll explain why I created this action, how it works, and how you may use it in your own markdown files in github. Motivation: Automating the Display of Achievements TryHackMe is a tool that many people in the cybersecurity field use to sharpen their abilities. However, manually updating badges or tracking your progress might be time-consuming, especially if you want to share this information on your GitHub page. I thought of a solution that would automate this process, allowing anyone to demonstrate their TryHackMe achievements with no effort. What Is the TryHackMe Badge Action Workflow? The TryHackMe Badge Action Workflow is a custom GitHub Action that refreshes and displays your TryHackMe badge on your GitHub profile or repository README page. This action allows you to maintain your achievements up to date without having to manually download, update, or commit your badge picture each time you acquire a new one. How It Works This action updates your GitHub profile or repository README with the new image and retrieves your most recent TryHackMe badge. This is a summary of the workflow’s functions: Image Fetching: The procedure begins with obtaining your TryHackMe badge’s most recent version from a given URL. Commit Changes: The downloaded image is then committed to your repository, guaranteeing that the most recent badge is always visible. README Update: Lastly, the action adds the updated badge image to your README file, allowing you to view your TryHackMe accomplishments in real time. Image Fetching: The procedure begins with obtaining your TryHackMe badge’s most recent version from a given URL. Commit Changes: The downloaded image is then committed to your repository, guaranteeing that the most recent badge is always visible. README Update: Lastly, the action adds the updated badge image to your README file, allowing you to view your TryHackMe accomplishments in real time. Usage Add the below script to your GitHub repository as a .yml file (e.g., update-tryhackme-badge.yml). Configure the action with the following inputs: GITHUB_TOKEN: Your GitHub Personal Access Token (required, set as a secret). image_path: The path to store the downloaded badge image (defaults to ./assets/tryhackme-badge.png). username: Your TryHackMe username (defaults to the value in a secret named THM_USERNAME). user_id: Your TryHackMe user ID (defaults to the value in a secret named THM_USER_ID). Add the below script to your GitHub repository as a .yml file (e.g., update-tryhackme-badge.yml). Add the below script to your GitHub repository as a .yml file (e.g., update-tryhackme-badge.yml). Configure the action with the following inputs: Configure the action with the following inputs: GITHUB_TOKEN: Your GitHub Personal Access Token (required, set as a secret). GITHUB_TOKEN: Your GitHub Personal Access Token (required, set as a secret). image_path: The path to store the downloaded badge image (defaults to ./assets/tryhackme-badge.png). image_path: The path to store the downloaded badge image (defaults to ./assets/tryhackme-badge.png). username: Your TryHackMe username (defaults to the value in a secret named THM_USERNAME). username: Your TryHackMe username (defaults to the value in a secret named THM_USERNAME). user_id: Your TryHackMe user ID (defaults to the value in a secret named THM_USER_ID). user_id: Your TryHackMe user ID (defaults to the value in a secret named THM_USER_ID). name: Update TryHackMe Badge

on:
  schedule:
    - cron: '0 0 * * *' # Runs every day at midnight
  workflow_dispatch: # Allows manual triggering

jobs:
  update-badge:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout repository
      uses: actions/checkout@v2

    - name: Fetch TryHackMe Badge
      uses: DhanushNehru/tryhackme-badge-action-workflow@v1.0
      with:
        image_path: './assets/tryhackme-badge.png'
        username: 'your_tryhackme_username'
        user_id: 'your_tryhackme_user_id'
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} name: Update TryHackMe Badge

on:
  schedule:
    - cron: '0 0 * * *' # Runs every day at midnight
  workflow_dispatch: # Allows manual triggering

jobs:
  update-badge:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout repository
      uses: actions/checkout@v2

    - name: Fetch TryHackMe Badge
      uses: DhanushNehru/tryhackme-badge-action-workflow@v1.0
      with:
        image_path: './assets/tryhackme-badge.png'
        username: 'your_tryhackme_username'
        user_id: 'your_tryhackme_user_id'
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} An easy-to-use yet effective solution for automating the display of your security accomplishments is the TryHackMe Badge Action Workflow. If you liked the project as a source of encouragement and support please give it a star ⭐️👇 https://github.com/marketplace/actions/tryhackme-badge-action-workflow?embedable=true https://github.com/marketplace/actions/tryhackme-badge-action-workflow?embedable=true I have used this in my Github README.md so feel free to check it out Github README.md Thanks for reading, please give a like as a sort of encouragement and also share this post in socials to show your extended support. Thanks for reading, please give a like as a sort of encouragement and also share this post in socials to show your extended support. Connect ⬇️ Connect ⬇️ X / Instagram / Github / Youtube / Newsletter / Discord X Instagram Github Youtube Newsletter Discord

Introducing the TryHackMe Badge Action Workflow: Automate Your Security Achievements Display

The Basics of Debian GNU/Linux

In the ever-evolving landscape of cyber threats, ransomware has proven to be one of the most notorious and damaging forms of attack. Among the various ransomware variants, RagnarLocker has garnered significant attention for its sophisticated tactics and impact on critical infrastructure sectors. In this blog post, we delve into the world of RagnarLocker ransomware, exploring its indicators of compromise (IOCs), technical intricacies, and the havoc it wreaks on targeted systems. 💥🔓 Ransomware Strikes Israeli Hospital: Ragnar_Locker Takes Responsibility 🏥 In a chilling turn of events, RagnarLocker’s menace struck closer to home as it targeted Mayanei Hayeshua Medical Center in Bnei Brak, Israel a few days ago. The ransomware attack wreaked havoc on the hospital’s administrative computer systems. The aftermath was devastating — medical treatments were disrupted, and the emergency room was compelled to redirect patients to other hospitals for urgent care. Emerging from the chaos, Ragnar_Locker took responsibility for this audacious attack. However, I’ve made a detailed investigation that yielded no signs of any leaked data or information on the group’s Onion site or Telegram group. This marks an unusual departure from the norm, where cybercriminals often flaunt their exploits on such platforms. This incident serves as a grim reminder of the vulnerabilities lurking in critical infrastructure sectors, with healthcare being no exception. It underscores the pressing need for fortified cybersecurity measures to shield essential services from such malicious onslaughts. As the investigation into this unsettling event progresses, the global cybersecurity community will undoubtedly be closely monitoring further developments. 🌐🔒🦠 The Emergence of RagnarLocker 🌄💻 The first traces of RagnarLocker ransomware came to light in April 2020, catching the attention of cybersecurity experts and organizations worldwide. Notably, the Federal Bureau of Investigation (FBI) issued an initial report detailing known IOCs associated with RagnarLocker. Over time, this ransomware variant has evolved, refining its obfuscation techniques and evading traditional detection methods. 🚀🔍 A Persistent Threat 🎯🌐 As of January 2022, the FBI has identified over 52 entities spanning across 10 critical infrastructure sectors affected by RagnarLocker ransomware. These sectors encompass critical manufacturing, energy, financial services, government, and information technology, underscoring the ransomware’s indiscriminate targeting. 💼🏢🔒 Technical Insights 🔍💡 Ransom Note and Identification: RagnarLocker is recognizable by the “.RGNR_<ID>” extension appended to encrypted files. The actors, self-identifying as “RAGNAR_LOCKER,” accompany this extension with a .txt ransom note, containing instructions for payment and data decryption. 💰🔐📄 Advanced Packing Techniques: The ransomware employs advanced techniques like VMProtect, UPX, and custom packing algorithms, adding layers of complexity to its execution. This obfuscation serves to complicate detection efforts. 🧩🔒🔍 Geographical Targeting: RagnarLocker uses the Windows API GetLocaleInfoW to identify the infected machine’s location. If the victim’s location falls within certain geopolitical areas, the ransomware terminates its processes, potentially hinting at the attackers’ geopolitical motivations. 🗺️🌍🔍 Selective Encryption: Instead of opting for a widespread encryption approach, RagnarLocker selectively encrypts files. It designates folders it won’t encrypt, allowing the system to function “normally” while encrypting files with valuable data. This method minimizes disruptions while maximizing the potential payout. 📁🔒🤖 Before encryption, Ragnar Locker spawns the following child processes: This system command deletes all shadow copies on the victim’s system, preventing data recovery by the victim : wmic.exe shadowcopy delete This system command also deletes shadow copies, preventing data recovery by the victim : vssadmin delete shadows /all /quiet This command launches Notepad.exe to show the ransom note to the victim. : notepad.exe [User path]\RGNR_AABBCCDD.txt After the ransom note is created, the actual file encryption process initiates using the __ __algorithm. It excludes files ending in . Salsa20 db, .sys, .dll, lnk, .msi, .drv and .exe. Neutralizing Defenses 🛡️🚫 Service Termination: RagnarLocker goes a step further in its attack strategy by identifying and terminating services commonly used by managed service providers (MSPs) for remote network administration. This further hampers recovery efforts. 💔🔧 Shadow Copy Deletion: Preventing victim recovery efforts, the ransomware erases Volume Shadow Copies, a mechanism used to restore files to previous states. The malware employs various methods, including >vssadmin delete shadows /all /quiet and >wmic.exe.shadowcopy.delete. 🗑️🔮 Conclusion 📢🔐 RagnarLocker ransomware presents a multifaceted threat that combines advanced technical techniques with an intricate attack strategy. Its evolution since its discovery in 2020, along with its targeting of critical infrastructure sectors, underscores the urgency of bolstering cybersecurity measures to defend against this and similar threats. As organizations and security experts continue to dissect RagnarLocker’s mechanisms, it becomes increasingly evident that the fight against ransomware demands a proactive and collaborative approach, where vigilance and innovation go hand in hand. 🛡️🌐🔒 References: https://www.timesofisrael.com/liveblog_entry/bnei-brak-hospital-bracing-for-release-of-sensitive-patient-information-after-hack/ https://www.ic3.gov/Media/News/2022/220307.pdf https://www.linkedin.com/showcase/dailydarkweb/ https://www.pcrisk.com/removal-guides/17018-ragnar-locker-ransomware Also published . here

Uncovering Ragnar_Locker Ransomware: Exploring IOCs

This article is part of my ↗️ where I share resources about building things. Join me :) Note: toolkit newsletters As a , I build things full-time, whether it’s a , or coaching founders to build tech products. For fun, I build , , , , , using my product builder venture newsletter micro-products AI article tool event app meal box app SaaS tracker sneaker app rapid MVP technique. Creating an MVP (minimum viable product) is a great way to test the viability of your startup idea and validate your assumptions before investing significant time and resources in building a full-fledged product. In this post, I'll share my frameworks for . Join my newsletter as I continue to provide valuable like , and . no-code MVP product guide series Validation Product Goals Product/Market Fit Step 1: Map out the users’ problems Don’t start with no-code tools (solution), start with the users (problem). Before you start thinking about no-code tools, figure out who has the problem by creating a user persona, what the users' problems are and how you can build something to meet their needs. You can investigate users’ problems through these methods: Interview potential users to uncover their frustrations Research about customer’s pain points through existing channels (forums, reddit, communities, keyword research) Create a Typeform survey and solicit feedback from a focus group Analyze and document feedback from potential customers Step 2: Map out the users’ journey Once you have a clear understanding of your users' problems, the next step is to map out the users’ journey. A user's journey represents the steps a user takes to achieve a specific goal and jobs to be done while interacting with your product. One easy way to figure out the touch points of your user is to create a customer journey map: There are four components in the user journey map. Let’s create a mapping for building a Ketogenic Recipe app. Example: Users want to find keto recipes using an app Stages: The stages of user path from point A to point B to point C. Hungry > Look up a recipe > Cook a meal User tasks: Actions a user will take at point A, point B and point C. Search for keto-friendly recipes > Browse recipes > Prepare meals Pain points: Problems facing a user at point A, point B and point C. Lack of varieties > Need to eliminate high-carbs foods > Limited recipe options Opportunities: List of possible solutions to solve the pain points at different stages. Answer a simple quiz > Personalized search > Provide recipe recommendations Step 3: Plan your MVP features Make a list of key functionality for your MVP. Brainstorm potential features that are critical to solving users’ pain points (in Step 2) in the most effective way. Example: Key functionality of a keto recipe app User login to the app using Google auth Separate recipes based on different categories Create advanced search and filter for unique dietary needs A list of keto recipes with ingredient lists and cooking instructions Ability to bookmark or save recipes How to keep your MVP lean: It's important to keep in mind that your MVP should be as simple as possible. So that you can launch quickly, test your assumptions and validate your app idea with minimum resources. Here’s my guiding principle: create a minimum set of features only include the essential features that are required to solve the user's problems solve one main problem (help users find keto recipes) and branch out later with wider scope (e.g. meal planner, community suggestions, personalize recipes) Step 4: Map out the technical requirement Based on the required MVP features, make a list of technical requirement that will help you in building the MVP functionality. Example Allow users to create an account, login, and access their saved recipes. User management: To display recipe item listing, upload recipe photo, description, ingredient lists, and tag food categories Database: Allow users to select specific dietary needs and search recipes by keywords Filter and search: This will help you determine what kinds of no-code tools to look for in the next step. Step 5: Evaluate no-code tools No-code tools are a great option for building MVPs because they allow you to create MVP features quickly and cheaply without any coding knowledge. In the previous step, you can see that we're looking for no-code platforms that will allow us to build out the features. user management, database, CMS, and advanced filter Therefore, when researching about no-code tools, you should look for tools that excel in these areas. Criteria for choosing no-code tools There’re other criteria to consider when choosing the right no-code tools. Let’s break it down: Look at all the no-code sample apps. Does the no-code tool allow you to create the functionality you want in your MVP? Are you building a mobile app or web-based app? Is the use case suitable? Can they be seamlessly integrated with other tools to enable you to stitch different components together (payment, automation, online forms, email list) and create a sufficient solution for your users? Does it allow me to integrate with other tools? Consider each tool's limitations. Is it easy to use? What does it do well and what does it not? Do you need to evaluate other alternative tools? How customizable is this tool? If you want to build your MVP further, how flexible is the no-code platform to accommodate the growth and expansion of your MVP? What is the cost involved in adding new features and functionalities as your MVP evolves? Feature expansion. [1] Find me on / 🔥 Twitter LinkedIn [2] Get my product-building teardown — 🚀 Join my newsletter Also Published Here

What App Features to Include in a No-Code MVP (5 Steps to Figure Out!)

Website scraping is much more than extracting content with some CSS selectors. We summarized years of expertise in this guide. With all these new tricks and ideas, you'll be able to scrape data reliably, faster, and more performant. And get some extra fields that you thought were not present. Prerequisites For the code to work, you will need . Some systems have it pre-installed. After that, install all the necessary libraries by running pip install. python3 installed pip install requests beautifulsoup4 pandas Getting the HTML from a URL is easy with the requests library. Then pass the content to BeautifulSoup, and we can start getting data and querying with selectors. We won't go into much detail. In short, you can use to get page elements and content. Some require a different syntax, but we'll discover that later. CSS selectors import requests from bs4 import BeautifulSoup response = requests.get("https://zenrows.com") soup = BeautifulSoup(response.content, 'html.parser') print(soup.title.string) # Web Data Automation Made Easy - ZenRows To avoid requesting the HTML every time, we can store it in an HTML file and load BeautifulSoup from there. For a simple demo, we can do this manually. An easy way to do that is to view the page source, copy and paste it into a file. It is essential to visit the page without being logged in, as a crawler would do. Getting the HTML in here might look like a simple task, but nothing farther from the truth. We won't cover it in this blog post, but it deserves a complete guide. Our advice is to use this static approach since many websites will redirect you to a login page after a few requests. Some others will show a captcha, and in the worst-case scenario, your IP will get banned. with open("test.html") as fp: soup = BeautifulSoup(fp, "html.parser") print(soup.title.string) # Web Data Automation Made Easy - ZenRows Once we load statically from a file, we can test as many times as possible without any networking or blocking problem. Explore before Coding Before we start coding, we have to understand the page's content and structure. For that, the easier way we know is to inspect the target page using a browser. . We will be using Chrome's DevTools, but other browsers have similar tools We can open any product page on Amazon, for example, and a quick look will show us the product's name, price, availability, and many other fields. Before copying all those selectors, we recommend taking a couple of minutes to look for hidden inputs, metadata, and network requests. Beware of doing this with Chrome DevTools or similar means. You will see the content once the Javascript and network requests have (maybe) modified it. It is tiresome, but sometimes we must explore the original HTML to avoid running Javascript. We won't need to run a headless browser - i.e., Puppeteer - if we find everything, saving time and memory consumption. Disclaimer: we won't include the URL request in the code snippets for every example. They all look like the first one. And remember, store an HTML file locally if you are going to test it several times. Hidden Inputs Hidden inputs allow developers to include input fields that end-users cannot see nor modify. Many forms use these to include internal IDs or security tokens. In , we can see that there are many more. Some will be available in other places or formats, but sometimes they are unique. Either way, hidden inputs' names tend to be more stable than classes. Amazon products Metadata While some content is visible via UI, it might be easier to extract using metadata. You can get view count in numeric format and publish date in YYYY-mm-dd format in a . Those two can be obtained with means from the visible part, but there is no need. Spending a few minutes doing these techniques pays off. YouTube video interactionCount = soup.find('meta', itemprop="interactionCount") print(interactionCount['content']) # 8566042 datePublished = soup.find('meta', itemprop="datePublished") print(datePublished['content']) # 2014-01-09 XHR Requests Some other websites decide to load an empty template and bring all the data via XHR requests. In those cases, checking just the original HTML won't be enough. We need to inspect the networking, specifically the XHR requests. That's the case for . Fill the form with any city and search. That will redirect you to a results page with a skeleton page while it performs some queries for the city you entered. Auction That forces us to use a headless browser that can execute Javascript and intercept network requests, but we'll see its upsides also. Sometimes you can call the XHR endpoint directly, but they usually require cookies or other authentication methods. Or they can instantly ban you since that is not a regular user path. Be careful. We struck gold. Take another look at the image. All the data you can have, already clean and formatted, is ready to be extracted. And then some more. Geolocation, internal IDs, numerical price with no format, year built etcetera. Recipes and Tricks to Extract Reliable Content Set aside your urges for a second. Getting everything with CSS selectors is an option, but there are many more. Take a look at all these, and then think again before using selectors directly. We are not saying those are bad and ours great. Don't get us wrong. We are trying to give you more tools and ideas. Then it will be your decision every time. Getting Internal Links Now, we will start using BeautifulSoup to get meaningful content. This library allows us to get content by IDs, classes, pseudo-selectors, and many more. We will only cover a small subset of its capabilities. This example will extract all the internal links from the page. For simplicity's sake, only links starting with a slash will be considered internal. In a completer case, we should check domain and subdomains. internalLinks = [ a.get('href') for a in soup.find_all('a') if a.get('href') and a.get('href').startswith('/')] print(internalLinks) Once we have all those links, we could deduplicate and queue them for future scraping. By doing it, we would be building a whole website crawler, not just for one page. Since that is an entirely different problem, we wanted to mention it and prepare a blog post to handle its usage and scalability. The number of pages to crawl can snowball. Just a note of caution: be prudent while running this automatically. You can get hundreds of links in a few seconds, which would result in too many requests to the same site. If not handled carefully, captchas or bans will probably apply. Extracting Social Links and Emails Another common scraping task is to extract social links and emails. There is no exact definition for "social links," so we'll obtain them based on domain. As for emails, there are two options: "mailto" links and checking the whole text. We will be using a for this demo. scraping test site This first snippet will get all the links, similar to the previous one. Then loop over all of them, checking if any social domains or "mailto" are present. In that case, add that URL to the list and finally print it. links = [a.get('href') for a in soup.find_all('a')] to_extract = ["facebook.com", "twitter.com", "mailto:"] social_links = [] for link in links: for social in to_extract: if link and social in link: social_links.append(link) print(social_links) #['mailto:****@webscraper.io', #'https://www.facebook.com/webscraperio/', #'https://twitter.com/webscraperio'] This second one is a bit more tricky if you are not familiar with regular expressions. In short, they will try to match any text given a search pattern. In this case, it will try to match some characters (mainly letters and numbers), followed by [@], then again chars - the domain - [dot] and finally two to four characters - Internet top-level domains or TLD. It will find, for example, . test@example.com Note that this regex is not a complete one because it won't match composed TLDs such as . co.uk We can run this expression in the entire content (HTML) or just the text. We use the HTML for completion, although we will duplicate the email since it's shown in the text and an href. emails = re.findall(   r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+.[A-Za-z]{2,4}",   str(soup)) print(emails) # ['@webscraper.io', '@webscraper.io'] Parse Tables Automatically HTML tables have been around forever, but they are still in use for displaying tabular content. We can take advantage of that since they are usually structured and well-formatted. Using as an example, we will extract all the values to an array and a pandas dataframe. It is a simple example, but you should manipulate all the data as if it came from a dataset. Wikipedia's List of best-selling albums We start by finding a table and looping through all the rows ("tr"). For each of them, find cells ("td" or "th"). The following lines will remove notes and collapsible content from Wikipedia tables, not strictly necessary. Then, append the cell's stripped text to the row and the row to the final output. Print the result to check that everything looks fine. table = soup.find("table", class_="sortable") output = [] for row in table.findAll("tr"): new_row = [] for cell in row.findAll(["td", "th"]): for sup in cell.findAll('sup'): sup.extract() for collapsible in cell.findAll( class_="mw-collapsible-content"): collapsible.extract() new_row.append(cell.get_text().strip()) output.append(new_row) print(output) #[ #['Artist', 'Album', 'Released', ...], #['Michael Jackson', 'Thriller', '1982', ...] #] Another way is to use and import directly the HTML, as shown below. It will handle everything for us: the first line will match the headers, and the rest will be inserted as content with the right type. returns an array, so we take the first item and then remove a column that has no content. pandas read_html Once into a dataframe, we could do any operation, like ordering by sales, since pandas converted some columns to numbers. Or sum all the claim sales. Not truly useful here, but you get the idea. import pandas as pd table_df = pd.read_html(str(table))[0] table_df = table_df.drop('Ref(s)', 1) print(table_df.columns) # ['Artist', 'Album', 'Released' ... print(table_df.dtypes) # ... Released int64 ... print(table_df['Claimed sales*'].sum()) # 422 print(table_df.loc[3]) #Artist			Pink Floyd #Album				The Dark Side of the Moon #Released			1973 #Genre				Progressive rock #Total certified copies...	24.4 #Claimed sales*		45 Extract from Metadata instead of HTML As seen before, there are ways to get essential data without relying on visual content. Let's see an example using . We'll try to get the actors.Easy, right? A one-liner will do. Netflix's The Witcher actors = soup.find(class_="item-starring").find( class_="title-data-info-item-list") print(actors.text.split(',')) #['Henry Cavill', 'Anya Chalotra', 'Freya Allan'] What if I told you that there are fourteen actors and actresses? Will you try to get 'em all? Do not scroll further if you want to try it by yourself. I'll wait. Nothing yet? Remember, there's more than meets the eye. You know three of them; search for those in the original HTML. To be honest, there's another place down below that shows the whole cast, but try to avoid it. Netflix includes a Schema.org snippet with the actor and actress list and many other data. As with the YouTube example, sometimes it is more convenient to use this approach. Dates, for example, are usually displayed in "machine-like" format, which is more helpful while scraping. import json ldJson = soup.find("script", type="application/ld+json") parsedJson = json.loads(ldJson.contents[0]) print([actor['name'] for actor in parsedJson['actors']]) #[... 'Jodhi May', 'MyAnna Buring', 'Joey Batey' ...] Some other times this is a practical approach if we don't want to render Javascript. We'll show an example using . They are known blockers. After visiting a few pages, you will be redirected to a login page. Be careful while scraping Instagram and use local HTML for testing. Instagram Billie Eilish's profile We'll cover how to avoid those blocks or redirects in a future post. Stay tuned! The usual way would be to search for a class, in our case "Y8-fY". We advise against using these classes since they will probably change. They look autogenerated. Many modern websites use this kind of CSS, and it gets generated for every change, which means that we cannot rely on them. Plan B: , right? It will work. But we need Javascript rendering for that since it is not present on the first load. As stated before, we should try to avoid that. "header ul > li" Take a look at the source HTML: title and description include followers, following, and post numbers. It might be a problem since they are in string format, but we can get over it. If we only want that data, we won't need a headless browser. Great! metaDescription = soup.find("meta", {'name': 'description'}) print(metaDescription['content']) #87.9m Followers, 0 Following, 493 Posts ... Hidden eCommerce Product Information Combining some of the techniques already seen, we aim to extract product information that is not visible. Our first example is a Shopify eCommerce, . Spigen Take a look on your own first if you want. Hint: look for the brand 🤐. We'll be able to extract it reliably, not from the product name nor the breadcrumbs, since we cannot say if they are always related. Did you find them? In this case, they use "itemprop" and include Product and Offer from . We could probably tell if a product is in stock by looking at the form or the "Add to cart" button. But there is no need, we can trust on . As for the brand, the same snippet as the one used for YouTube but changing the property name to "brand." schema.org itemprop="availability" brand = soup.find('meta', itemprop="brand") print(brand['content']) # Tesla Another Shopify example: . We want to extract the rating count and average, accessible in the HTML but somewhat hidden. The average rating is hidden from view using CSS. nomz There's a screen reader only tag with the average and the count near it. Those two include text, not a big deal. But we know we can do better. It is an easy one if you inspect the source. The Product schema will be the first thing you see. Applying the same knowledge from the Netflix example, get the first "ld+json" block, parse the JSON, and all the content will be available! import json ldJson = soup.find("script", type="application/ld+json") parsedJson = json.loads(ldJson.contents[0]) print(parsedJson["aggregateRating"]["ratingValue"]) # 4.9 print(parsedJson["aggregateRating"]["reviewCount"]) # 57 print(parsedJson["weight"]) # 0.492kg -> extra, not visible in UI Last but not least, we will take advantage of data attributes, which are also common in eCommerce. While inspecting , we can see that every product has several data points that will come in handy. Price in numeric format, ID, product name, and category. We have there all the data we might need. Marucci Sports Wood Bats products = [] cards = soup.find_all(class_="card") for card in cards:   products.append({     'id': card.get('data-entity-id'),     'name': card.get('data-name'),     'category': card.get('data-product-category'),     'price': card.get('data-product-price')   }) print(products) # [ #   { #     "category": "Wood Bats, Wood Bats/Professional Cuts", #     "id": "1945", #     "name": "6 Bat USA Professional Cut Bundle", #     "price": "579.99" #   }, #   { #     "category": "Wood Bats, Wood Bats/Pro Model", #     "id": "1804", #     "name": "M-71 Pro Model", #     "price": "159.99" #   }, #   ... # ] Remaining Obstacles All right! You got all the data from that page. Now you have to replicate it to a second and then a third. Scale is important. And so is not getting banned. But you also have to convert this data and store it: CSV files or databases, whatever you need. Nested fields are not easy to export to any of those formats. We already took enough of your time. Take in all this new information, use it in your everyday work. Meanwhile, we'll be working on the following guides to overcoming all these obstacles! Conclusion We'd like you to go with three lessons: CSS selectors are good, but there are other options. Some content is hidden - or not present - but accessible via metadata. Try to avoid loading Javascript and headless browsers to boost performance. Each of these has upsides and downsides, different approaches, and many, many alternatives. Writing a complete guide would be, well, a long book, not a blog post. if you know any more website scraping tricks or have doubts about applying them. Contact us Remember, we covered scraping, but there is much more: crawling, avoiding being blocked, converting and storing the content, scaling the infrastructure, and more. Stay tuned! And if you liked the content, please share it! Also published on: https://www.zenrows.com/blog/mastering-web-scraping-in-python-from-zero-to-hero

How to Master Web Scraping in Python: From Zero to Hero

The last few years have been a very riveting time. We witness a global integration of different spheres. One of the most prominent examples is AI. This technology is already being applied in medicine, education, design and of course, marketing. Why is this happening? Mainly because the new reality demands new ideas from businesses. As a result, marketers are forced to actively seek insights and unconventional solutions to better understand their customers/users and interact with them more effectively and faster than competitors. Over the past few years, the business has shifted its focus to digitalization, and 2020 has been a tremendous trigger to accelerate the process. Now the companies' attention is focused on creating their own IT products. And customer are at an all-time high. expectations This has led to companies being in a constant, tight race for user loyalty and love. Those who can offer the best user experience lead the race. It is a big challenge for marketers as well — they need to dig deeper into the product, user path, preferences, and market changes. And to do that, they need insights to percept business from different angles and push the boundaries of ideas. One way to find such an idea is to look at the business through the eyes of someone from another profession, such as a tester. Who better to understand the user than him, testing different variations of behavior over and over again? A marketer and a tester? At first sight, you can think that they have absolutely different areas of responsibility that do not overlap. Really, what good can testers do in marketing if they are not immersed in the business and marketing processes? However, in recent years, testing has become deeply integrated into business. What has changed? First, I purposely put the emphasis on QA in the title of the article. Testers and QA are already inseparable concepts. These experts monitor the quality of the product from all sides (functional, logical, and usability), immersing themselves in the business processes and business logic of the product. Secondly, they try on the masks of your product's TA ( ). Therefore, they have a good understanding of what will appeal to the user and make good business sense. think of the character method at least A tester can be a marketer's right hand, as the key to success in business promotion is understanding your target audience. As you can see, marketers and testers do have something in common. At a minimum, interest in keeping the user happy with the product so that loyalty grows. At a maximum, an understanding of what it takes to do it. If they have a common goal, then they can help each other to achieve it. I will show you that such an unusual integration hides a lot of insights and benefits for business and I give you some examples from my testing and QA agency. So, what can testers do for marketers? And how does this unusual alliance help business? Improve the user experience The user experience is one of the main things that affects product loyalty and sales. According to a report, one in three customers abandons a brand after just one bad experience, and 92% leave completely after two or three negative interactions. To improve the interaction with the product (website or app) and increase conversions, you need to understand if there is a problem with functionality or usability. And also analyze why users leave, do not take the targeted action, and what exactly they do not like. PwC Of course, there are many different useful tools that marketers can use on their own. For example, the simplest and most accessible is Google analytics. But there is a crucial point. Just like in the development phase, in product analytics you need the perspective not of someone who is immersed in the specifics of the product, but of someone who will be using it. What can a tester tell you: How the user interacts with the product; Whether the user is comfortable, whether he is satisfied with everything in functionality or usability; Whether problems arise at any stage of interaction with the product; What prevents the user from taking the targeted action (in terms of functionality and usability). The tester has a great background, fully immerses himself in the user experience, asks a lot of questions, and endlessly tests possible variations in user behavior. This is why the tester understands the user's logic well. Case study: Superfluous forms and elements are frequent enemies of conversions. The more actions you force the user to take, the lower the chance of him reaching the end. We had a case: a client had a low conversion rate at the stage of purchasing in the online store. A portion of users left the page when they were about to make a purchase. When we started testing, we realized that the problem was with unnecessary fields. For example, the «patronymic» field was mandatory. We made all not really important fields voluntary ones. After the release, they conversion was up. So, we continued the experiment and removed all unnecessary fields to fill in and merged some fields into one. Build processes and make them more efficient I think everyone who has worked with product teams understands the importance of a well-functioning team and inter-team communication processes. And really, it is not just a story about IT teams. Weak processes affect the entire chain - from the moment an idea is conceived to its implementation and launch. And in the end, this is reflected in the quality of the product and user loyalty. Statistically, 72% of clients will share a positive experience with 6 or more people. If customers are dissatisfied, 13% of them will share their with 15 or more people. The processes are handled by a specially trained QA rather than a tester. experience They can put processes not only between teams but for example between IT and marketing and even between accounting departments. These professionals look at a company like clockwork and see where the failure has occurred. What you can learn from QA: If weak processes affect product quality and user loyalty; What points in the work processes need to be addressed and what changes need to be made to improve the end result; How to communicate with other departments so that you do not miss important details and improve your product performance. Case study: Basically, processes are how tasks are distributed and how information is communicated; how departments and teams interact, etc. Our client had a high percentage of customer service calls. For several months, our QA has been putting teamwork processes in place and selecting flexible methods of interaction between employees. The result was stunning — after only two months the number of complaints had already decreased by 4 times. Conduct a competitive analysis Analytics are the eyes and ears of the marketer. Without it, it is impossible to create a plan of promotion and brand positioning in the market. And of course, competitive analysis is important in analytics. It is necessary to study not only what competitors are doing, but also why users like their products, or vice versa — why they abandon it. Again, you may ask the tester about it. What you can find out from the tester; How users interact with a competitor's product; What the user path is; Does your competitor have cool features that your product does not; What can be improved to outperform a competitor's product. Case study: We often get requests «we need competitive analysis of websites or applications». We look at the functionality of all the sites, map the differences and thus find features or bugs in the competitors. For example, our client, a marketplace, analyzed competitors' products and created the same functionality. And asked us to do an analysis and see if they took everything into account. When we started checking, we saw that there was an inconspicuous section on a competitor's site that hid a useful feature with the ability to print orders and receipts. This allowed our client, firstly, to implement this feature in its functionality. Secondly, do not repeat the competitor’s and do not hide it in inconspicuous sections. I hope that I was able to show you real examples of how a seemingly strange union of two different professions can become an effective tool for business. It is important not to be afraid to look for insights where there previously seemed to be none. And at the same time constantly communicate with colleagues from different fields, exchanging experiences and observations. After all, this is where the great potential for business development and growth lies.

QA + Marketing: Can Testers Help Marketers and Vice Versa?

Continuing my personal journey into learning more about , I wanted to create a graph visualization of data stored in a graph database.  Graph visualization (or link analysis) presents data as a network of entities that are classified as nodes and links. To illustrate, consider this very simple network diagram: Dgraph Slash GraphQL While not a perfect example, one can understand the relationships between various services (nodes) and their inner-connectivity (links). This means the X service relies on the Y service to meet the needs of the business. However, what most may not realize is the additional dependency of the Z service, which is easily recognized by this illustration. For this article, I wanted to build a solution that can dynamically create a graph visualization. Taking this approach, I will be able to simply alter the input source to retrieve an entirely different set of graph data to process and analyze. The Approach Instead of mocking up data in a Spring Boot application (as noted in the " " article), I set a goal to utilize actual data for this article. Tracking the Worst Sci-Fi Movies With Angular and Slash GraphQL From my research, I concluded that the key to a building graph visualization is to have a data set that contains various relationships. Relationships that are not predictable and driven by uncontrolled sources. The first data source that came to mind was Twitter. After retrieving data using the Twitter API, the JSON-based data would be loaded into a Dgraph Slash GraphQL database using a somewhat simple Python program and a schema that represents the tweets and users captured by and uploaded into Slash GraphQL. Using the and the graph visualization library, the resulting data will be graphed to visually represent the nodes and links related to the #NeilPeart hashtag. The illustration below summarizes my approach: twarc Angular CLI ngx-graph Retrieving Data From Twitter Using `twarc` While I have maintained a semi-active Twitter account ( ) for almost nine years, I visited the to create a project called "Dgraph" and an application called "DgraphIntegration". This step was necessary in order to make API calls against the Twitter service. @johnjvester Twitter Developer Portal The solution (by DocNow) allows for Twitter data to be retrieved from the Twitter API and returned in an easy-to-use, line-oriented, JSON format. The twarc command line tool was written and designed to work with the Python programming language and is easily configured using the command and supplying the following credential values from the "DgraphIntegration" application: twarc twarc configure CONSUMER_KEY CONSUMER_SECRET ACCESS_TOKEN ACCESS_TOKEN_SECRET With the death of percussionist/lyricist , I performed a search for hashtags that continue to reference this wonderfully-departed soul. The following search command was utilized with twarc: Neil Peart twarc search #NeilPeart > tweets.jsonl Below is one example of the thousands of search results that were retrieved via the twarc search: { : , : , : { : , : }
} "content" "It’s been one year since he passed, but the music lives on... he also probably wouldn’t have liked what was going on in the word. Keep resting easy, Neil #NeilPeart https://t.co/pTidwTYsxG" "tweet_id" "1347203390560940035" "user" "screen_name" "Austin Miller" "handle" "AMiller1397" Preparing Dgraph Slash GraphQL Starting in September 2020, Dgraph has offered a fully managed backend service, called Slash GraphQL. Along with a hosted graph database instance, there is also a RESTful interface. This functionality, combined with 10,000 free credits for API use, provides the perfect target data store for the #NeilPeart data that I wish to graph. The first step was to create a new backend instance, which I called : tweet-graph Next, I created a simple schema for the data I wanted to graph: type User { : ! @id @search(by: [hash])
  screen_name: ! @search(by: [fulltext])
}

type Tweet { : ! @id @search(by: [hash])
  content: !
  user: User
}

type Configuration { : ID
  search_string: !
} handle String String tweet_id String String id String The and types house all of the data displayed in the JSON example above. The type will be used by the Angular client to display the search string utilized for the graph data. User Tweet Configuration Loading Data into Slash GraphQL Using Python Two Python programs will be utilized to process the JSON data extracted from Twitter using twarc: convert - processes the JSON data to identify any Twitter mentions to another user upload - prepares and performs the upload of JSON data into Slash GraphQL The core logic for this example lives in the upload program, which executes the following base code: data = {}
users = {}

gather_tweets_by_user()

search_string = os.getenv( )
print(search_string)

upload_to_slash(create_configuration_query(search_string)) handle data:
    print( )
    upload_to_slash(create_add_tweets_query(users[handle], data[handle])) 'TWARC_SEARCH_STRING' for in "=====" The organizes the Twitter data into the and objects. gather_tweets_by_user() data users The stores the search that was performed into Slash GraphQL for use by the Angular client upload_to_slash(create_configuration_query(search_string)) The loop processes the and objects, uploading each record into Slash GraphQL using for data user upload_to_slash(create_add_tweets_query(users[handle], data[handle])) Once the program finishes, you can execute the following query from the API Explorer in Slash GraphQL: query MyQuery {
  queryTweet {
    content
    tweet_id
    user {
      screen_name
      handle
    }
  }
}

query MyQuery {
  queryConfiguration {
    search_string
  }
} Using `ngx-graph` With Angular CLI The Angular CLI was used to create a simple Angular application. In fact, the base component will be expanded for use by , which was installed using the following command: ngx-graph npm install @swimlane/ngx-graph --save Here is the working AppModule for the application: @NgModule({ : [
    AppComponent
  ], : [
    BrowserModule,
    BrowserAnimationsModule,
    AppRoutingModule,
    HttpClientModule,
    NgxGraphModule,
    NgxChartsModule,
    NgxSpinnerModule
  ], : [CUSTOM_ELEMENTS_SCHEMA], : [], : [AppComponent]
}) { } declarations imports schemas providers bootstrap export class AppModule In order to access data from Slash GraphQL, the following method was added to the GraphQlService in Angular: allTweets:string = + + + + + + + + + ;

getTweetData() { .http.get<QueryResult>( .baseUrl + + .allTweets).pipe(
    tap(),
    catchError( { ErrorUtils.errorHandler(err)
    }));
} 'query MyQuery {' '  queryTweet {' '    content' '    tweet_id' '    user {' '      screen_name' '      handle' '    }' '  }' '}' return this this '?query=' this => err return Preparing Slash GraphQL to Work With `ngx-graph` The data in Slash GraphQL must be modified in order to work with the framework. As a result, a ConversionService was added to the Angular client, which performed the following tasks: ngx-graph createGraphPayload(queryResult: QueryResult): GraphPayload { graphPayload: GraphPayload = GraphPayload(); (queryResult) { (queryResult.data && queryResult.data.queryTweet) { tweetList: QueryTweet[] = queryResult.data.queryTweet;

      tweetList.forEach( { tweetNode:GraphNode = .getTweetNode(queryTweet, graphPayload); userNode:GraphNode = .getUserNode(queryTweet, graphPayload); (tweetNode && userNode) { graphEdge:GraphEdge = GraphEdge();
          graphEdge.id = ConversionService.createRandomId(); (tweetNode.label.substring( , ) === ) {
            graphEdge.label = ;
          } {
            graphEdge.label = ;
          }

          graphEdge.source = userNode.id;
          graphEdge.target = tweetNode.id;
          graphPayload.links.push(graphEdge);
        }
      });
    }
  } .log( , graphPayload); graphPayload;
} let new if if let ( ) => queryTweet let this let this if let new if 0 2 'RT' 'retweet' else 'tweet' console 'graphPayload' return The resulting structure contains the following object hierarchy: {
  links: GraphEdge[] = [];
  nodes: GraphNode[] = [];
} {
  id: string;
  label: string;
  source: string;
  target: string;
} {
  id: string;
  label: string;
  twitter_uri: string;
} export class GraphPayload export class GraphEdge implements Edge export class GraphNode implements Node While this work could have been completed as part of the load into Slash GraphQL, I wanted to keep the original source data in a format that could be used by other processes and not be proprietary to . ngx-graph Configuring the Angular View When the Angular client starts, the following method will fire, which will show a spinner while the data is processing. Then, it will display the graphical representation of the data once Slash GraphQL has provided the data and the ConversionService has finished processing the data: OnInit ngOnInit() { .spinner.show(); .graphQlService.getConfigurationData().subscribe( { (configs) { .filterValue = configs.data.queryConfiguration[ ].search_string; .graphQlService.getTweetData().subscribe( { (data) { queryResult: QueryResult = data; .graphPayload = .conversionService.createGraphPayload(queryResult); .fitGraph(); .showData = ;
        }
      }, (error) => { .error( , error);
      }).add( { .spinner.hide();
      });
    }
  }, (error) => { .error( , error);
  }).add( { .spinner.hide();
  });
} this this => configs if this 0 this => data if let this this this this true console 'error' => () this console 'error' => () this On the template side, the following tags were employed: ngx <ngx-graph *ngIf= = layout= [view]= [showMiniMap]= [zoomToFit$]= [links]= [nodes]= > <svg:marker id="arrow" viewBox="0 -5 10 10" refX="8" refY="0" markerWidth="4" markerHeight="4" orient="auto">
      <svg:path d="M0,-5L10,0L0,5" class="arrow-head" />
    </svg:marker>
  </ng-template>

  <ng-template #nodeTemplate let-node>
    <svg:g class="node" (click)="clickNode(node)">
      <svg:rect
        [attr.width]="node.dimension.width"
        [attr.height]="node.dimension.height"
        [attr.fill]="node.data.color"
      />
      <svg:text alignment-baseline="central" [attr.x]="10" [attr.y]="node.dimension.height / 2">
        {{node.label}}
      </svg:text>
    </svg:g>
  </ng-template>

  <ng-template #linkTemplate let-link>
    <svg:g class="edge">
      <svg:path class="line" stroke-width="2" marker-end="url(#arrow)"></svg:path>
      <svg:text class="edge-label" text-anchor="middle">
        <textPath
          class="text-path"
          [attr.href]="'#' + link.id"
          [style.dominant-baseline]="link.dominantBaseline"
          startOffset="50%"
        >
          {{link.label}}
        </textPath>
      </svg:text>
    </svg:g>
  </ng-template>

</ngx-graph> "showData" class "chart-container" "dagre" "[1720, 768]" "false" "zoomToFit$" "graphPayload.links" "graphPayload.nodes" < # > ng-template defsTemplate The tags not only provide a richer presentation of the data but also introduce the ability to click on a given node and see the original tweet in a new browser window. ng-template Running the Angular Client With the Angular client running, you can retrieve the data from the Slash GraphQL by navigating to the application. You will then see a  user experience similar to below: It is possible to zoom into this view and even rearrange the nodes to better comprehend the result set. Please note: For those who are not fond of the "dagre" layout, you can adjust the property to another in . ngx-graph.layout graph layout option ngx-graph When the end-user clicks a given node, the original message in Twitter displays in a new browser window: Conclusion A fully-functional Twitter Graph was created using the following frameworks and services: Twitter API and Developer Application twarc and custom Python code Dgraph Slash GraphQL Angular CLI and ngx-graph In a matter of steps, you can analyze Twitter search results graphically, which will likely expose links and nodes that are not apparent through any other data analysis efforts. This is similar to the network example in the introduction of this article that exposed a dependency on the Z service. If you are interested in the full source code for the Angular application, including the Python import programs referenced above, please visit the following repository on GitLab: https://gitlab.com/johnjvester/tweet-graph-client Have a really great day! Also published at https://dzone.com/articles/creating-a-twitter-graph-using-slash-graphql

How To Make a Twitter Graph with Slash GraphQL

For starters, let’s define our concepts. What is a lead? A lead is a user who is interested in your product: they have signed up on your website and subscribed to your emails. What is the difference between leads and other users? Leads have already shown interest in your product and shared their contact information with you. What is lead generation? is a process of turning your users into leads. Lead generation This is how a user’s path looks like: first, they visit your website, blog or social network page and get acquainted with your product. At this stage, they only get you to know, so most likely they are not ready to buy from you yet. Say, they like your product and decide to share with you their contact. You have a lead now! Yay! Actually, there’s not much to celebrate yet, as What to do next? 50% of leads are not ready to buy from you right now, even more so, they are not ready to buy from you later as well. Next comes a hard and treacherous path of nurturing your leads. What is lead nurturing? is a process of preparing and stimulating potential users to purchase. Lead nurturing How to help your lead make a purchase? How not to miss opportunities to do so? There are several ways to complete these tasks, but let’s not move far away from  the topic at hand (which is email campaigns). What are the pros of email marketing? With a user’s email address, you get an opportunity to start personally communicating with this user and building a long-term relationship with them. With the right timing and message for your user segment, your nurture campaign will be even more effective. Sounds enticing, but it gets harder the moment you get closer to the actual implementation. What kind of emails do you need to send to your users? What kind of emails will they be interested in? You can choose subjects and settings for your nurture campaigns based on the chosen segment and other attributes. We are on a mission to help you. So here are the keystone kinds of nurture campaigns for every step of the marketing funnel. Just in case you run out of inspiration Types of nurture campaigns The goal of every email campaign listed below is in stimulating users to make a purchase. Some campaigns will encourage users to buy something and other campaigns will build trusting relationships between you and your users. The end result is basically the same in every case, however. Discount Discounts will help you motivate your users to make a purchase. The goals of your discount campaign may vary based on the user segment you’ve chosen for it. If it’s a potential customer, a discount you offer them may be a friendly reminder to purchase. If you’re giving a discount to your active customers, this offer may be used to strengthen their brand loyalty. Promotion campaign Promotion campaign is a direct call to buy, signup or subscribe. An effective email campaign will help you increase sales. This kind of campaigns will work with both your potential customers and the active ones. For a promotion campaign to give you a sales boost, it should have a nice design and be aimed at the right audience, segmented by location, age, gender etc. Reminder Reminder emails will warn customers who have forgotten about their trial period, subscription or discount coming to an end. The goal of an email like this is to get your customers back to the purchase before you lose them completely. An example of such email in Dashly Left the website at the payment stage Sometimes users change their mind and don’t finish their purchase or don’t fill a leadform on your website. Probably, they misunderstood something and need some help? Emails of this kind are sent automatically and may boost your conversion on the same traffic. This is what Statista sends to users who left the website: Feedback Such emails will gather feedback from your customers, and the feedback will help you increase the quality of your product and services. Besides, asking customers what they think about your product or service may help you increase their brand loyalty. Feedback emails are usually the triggered ones, these are sent right after completing a payment or some time after subscribing to your platform. The feedback you’ve gathered can be used in other nurture emails to increase brand loyalty. Updates Update emails are aimed at your active customers. The goal of such emails is to notify users about changes in your product. These also nurture both cold and warm leads because you’re telling them that your product is constantly improved. That’s how an Instagram planning app notified about its Instagram Stories update: Content Content campaigns are not about selling right here and right now, their goal is to build trusting relationships with users by sharing useful information like articles, videos and guides. Unlike promotion campaigns, content campaigns contain no direct call to purchase. With emails of this kind, companies demonstrate their expertise and implicitly push a sales boost. Announcement Announcement campaigns are basically any emails notifying about upcoming events, new projects, schedule changes, etc. Email campaigns of this kind are aimed at active users interested in such notifications. For example, we use these to remind our users about webinars and encourage them to visit our website again – probably, there’s something new? This way Miro tells about the full agenda release: Welcome email Welcome email are sent to users after completing their signup. Subscribers open welcome emails more often than users of other segments. The goal of welcome  emails is to give users the best possible presentation of your company. At the same time, it’s important not to overload users with information in a welcome email. Define the subject of that campaign based on your marketing goals, and don’t forget to make it nice and discreet enough too! Greeting Greeting emails will help you increase your users’ loyalty and reactivate them. There are two types of occasions to send a greeting email: greeting a user on their birthday and greeting them on any other significant date such as a commonly celebrated holiday or even “You’ve been with us for more than 30 days!”. To send birthday greeting emails, your customer base must contain not only email addresses of your customers, but their birthday dates as well. The task does get harder because of this, but the CTR of this greeting email will be much higher than that of an impersonal Christmas greeting email. Reactivation Reactivation emails are aimed at users who have forgotten about you. Emails of this kind reactivate users who don’t buy anything from you. These emails may contain a discount or a free service license. The goal of a reactivation email campaign is to get your customer to purchase from you again and to find out why they had left you in the first place. That’s how Coursera (an online learning platform) is trying to get me to learn with them again: Of course, the potential of email campaigns is not limited by this list alone. We’ve listed the kinds of campaigns that we think will be of the most value to you! Ultimately, it’s up to you to decide what email campaigns to launch. Just make sure they meet your marketing goals. Pay attention to the right segmentation as well. Don’t just stop at choosing what kind of a campaign to make. Without the right segmentation, you might end up sending your emails to the wrong audience. Without automation, your marketing specialist might waste a ton of time sending the emails. That’s why we decided to tell you about how to make it easier for your marketing team, while also increasing the quality of your campaign and helping you make sure your emails hit the right segment. The following guide will help you increase your sales and LTV metrics. How to make your email campaigns better Qualify Qualification builds your target audience. It also lets you rank them by their interests. With that information, you’ll be able to segment your customers easily, without a risk of wasting time and resources on those who are out of your target audience. Qualification is usually done via pop-up windows or sales calls. The questions usually depend on your product and your target audience. Dashly has ready-made qualification pop-ups like. They enable us to segment audience and send relevant messages to particular groups of people. Segment Segmentation lets you divide your leads and users into groups of matching features (such as gender, age, geography etc.) so that you can offer more personal and profitable deals to your customers. By segmenting, you can get as close to your customers as possible, sending them the content they will be 100% happy to see. You can use marketing automation, pop-ups and email campaigns to segment your customers. Try segmenting outside of the standard segments. Your product defines certain customer groups that may be of a strategic value to you. Automate Automated email campaign is set according to certain parameters (like time, date, event, segment etc.) that automatically fire after getting triggered. Automation makes emailing job much easier. What you can do by automating: Plan email campaigns; Send emails to special customer segments; Send triggered emails; Create email sequences. How do you automate? Using email marketing platforms, of course. With marketing automation, you can also track your visitors’ behaviour. You can see who reads your email and who clicks a link in it. Personalize Personalization is a must of all email campaigns nowadays. It lets you drastically improve your email open rate too. Another advantage of personalization is that it won’t take you too long to get it work. With automation and segmentation, you can set your personalized email campaigns that will be sent to specific user segments at a particular moment. Other than that, you’ll only need to make your emails look nice and make sure they deliver the good content. Personalized notification from Hackernoon: There you have it, a long list of nurture email campaigns for practically all occasions. Just remember that there’s no point in using every single one of them at the same time. Try starting a couple of basic campaigns, perfecting their quality. Choose what user segments to send emails to, automate and personalize them, and don’t forget to work on their design too.

12 Ways You Can Use Email to Nurture Leads to Conversion

An autofunnel is the user’s path from getting acquainted with making a purchase where you accompany him or her with pre-configured trigger messages. The important advantage lies precisely in the “auto” thing: you configure the funnel once and forever (for a long time). The implementation of the autofunnel requires virtually no human involvement. You can do this using Dashly.io Usually, the path of the customer is as follows: a visitor — a cold lead — a hot lead — a customer — a repeat customer. An autofunnel saves your resources while increasing the number of customers. That is, with its help you “handle” more potential customers without spending time on staff and additional budget. What does an autofunnel consist of? An autofunnel consists of several sequential stages; at each stage, you make an offer to the user which leads him or her to the ultimate goal — making a purchase. this is a small free product that will attract the potential customer. The task of a lead magnet is to generate leads, so this product is usually offered in exchange for an email or a phone number. Stage 1 – Lead Magnet: for copywriting agency, this is a check-list for creating selling headlines; for online art courses — lecture notes on dadaism. Example of a lead magnet: A good lead magnet can be identified by three characteristics: it solves the user’s problem; it’s clear from its description what is waiting for the user; it can be obtained immediately. In order to understand what will become your lead magnet, analyze the search queries’ statistics on your subject or explore the thematic forums. You need to understand the challenges your potential customers face on a daily basis. You don’t have to try to solve a global problem — your core product can handle it. A lead magnet is designed to meet a small but urgent need. You can buy windows in different ways this is a cheap product that’s interesting to users. It is attractive thanks to its price; potential customers get the opportunity to test your product and almost not spend. Thanks to the tripwire, a lead becomes a customer. Stage 2 – Tripwire: selling the cheapest plan for $2 instead of $20 for those who submit their email to get a lead magnet. Example of a tripwire: Importantly, be prepared for that the tripwire won’t make you any profit. Its goal is to attract users at a low cost and allow them to get to know you, turn a lead into a paying (although a little) customer. A good way to define a tripwire is to take the main product, separate one part from it and design it as an independent product. this is the main product around which your business is built and for which you need an autofunnel. It is for the purchase of the core product that a lead magnet and a tripwire prepared a user for. Stage 3 – Core product: for online art courses, this is the key course with feedback and a certificate. Example of the core product: are additional products and services that you offer along with the core product. Profit Maximizers Example: a customer has paid for online training and you offer them to buy an extended reference list. From Theory to Practice To build an autofunnel, you need to configure and run all the items that we listed above. It is much easier and more interesting to figure everything out on an example, so let’s imagine that we have an online school of confectioners. There are many training options: small workshops, webinars, occasional offline meetings, but your main product is a three-month course consisting of several modules. The course is quite expensive, but it is delivered by the country’s best confectioners, and in the end, you get a certificate confirmed by the confectioners’ association. What kind of autofunnel will help you increase sales and how to configure it? The Principles of Designing an Autofunnel 1. Create a Website or a Landing Page Before configuring an autofunnel, you need to prepare the basis, which is a website or a landing page where an autofunnel will work. Today, even a non-professional can generate their own single-page website; the main thing is to understand the basic principles of branding and select the appropriate service. Connect Google Analytics counter to your website. It is better to use both services to get more complete reports. The implementation of counters is free and it will not take you much time. With their help, you will be able to analyze user behavior on your website and fine-tune the autofunnel. Integrate the payment service into your website to open sales, and a CRM system to collect customer data and store it in one place. 2. Define your core product It’s quite simple: you probably know your core product. During the preparation stage of the autofunnel, it is worth checking the quality of the core product because you are making an effort exactly to sell it. The core product is the most expensive in the funnel so it is important that it contains the value for the customer. Let’s get back to our online school. Probably, clients more often sign up for and buy inexpensive workshops: there is no need to find time and budget for them. But the big course is your core product because it has the greatest value. The core product of an online school is a great course3. Make up and create a lead magnet There are lots of lead magnet options. Here are some of the most popular ones: checklist; guide / manual; cheat sheet; video lesson;i infographics; toolkit. A free workshop on meringue baking can become a lead magnet for the online confectionery school; a user submits their name and email in a special form and gets a link to a video lesion via email letter. 4. Create a tripwire A tripwire should be: simple and easy to use; affordably priced; encouraging to buy the core product (“if I’m getting this cool thing for a penny, then what am I going to get for the full price?”). The cost of a tripwire is determined by your niche. For example, if you sell the subscription service for $500 per month, offer a one-time consultation for $5. For our online confectionery school, a small course of five lessons on cakes will be a great tripwire. 5. Define Profit Maximizers Profit maximizers are products that complement the core product and help you increase your average order value. Profit maximizers are easy to identify; they are usually inexpensive add-on products such as smartphone safety glass. As a profit maximizer, a baking starter kit can be offered to buyers of large confectionery courses. 6. Tie it all together, come up with an autofunnel script. Once you have defined all the components of the autofunnel, the most interesting part begins: you need to connect them together and to declare the customer journey. It is very convenient to make up the scenario of an autofunnel using . Miro Let’s imagine a potential client of the confectionery school: a woman aged 30-40, married and on her maternity leave. She loves to cook and doesn’t like her old job in the office to which she is about to return. One day, she looks for a simple dessert recipe on the Internet and finds a free meringue baking workshop. A workshop is much better than simple recipes with pictures that she has seen before, so she does not hesitate to submit her email and gets a video lesson in the letter. From time to time, our heroine receives a newsletter with interesting recipes and spends a lot of time on the website. One day she sees a pop-up with an offer to take a small course on baking custard cakes. The course is paid, but it is very inexpensive, especially thanks to recipes from the mailing list that she has already managed to get carried away with baking. The woman pays for the course and is looking forward to the start. The small course was very useful; the woman learned 7 ways to prepare a cream. During this time, she became a regular visitor on the school’s website and has already found out about a large three-month course. She had no reason for a long doubt: the woman bought the training (and together with it she ordered a confectionery starter kit), successfully completed it, started baking cakes on request and never returned to the office again. A visitor comes to a website – Registration for a free workshop – Subscription to a newsletter – Pop-up with an invitation to a small course – Taking a small course – Subscription to a large course 7. Configure the autofunnel script and run it. Once the script has been made up, all you need is to solve the technical issues and get started. So, in order to implement the script and run the autofunnel we will need the following: A : we are going to offer a lead magnet in exchange of an email in it; lead generation pop-up with a link to a lead magnet; Welcome letter A that will maintain interest between the funnel stages; chain of content letters A pop-up with to a small course; an invitation A pop-up with . related products Each of these messages can be easily configured in Dashly. For example, a lead generation pop-up takes 7 minutes to be ready, and a welcome letter takes 10. Thus, the technical setup of the autofunnel will take you a couple of hours (along with the selection of images and copywriting). Then you’ll have to run it and enjoy the result:) It’s going to work out! Every business has its own strategy, and there is no universal formula for an ideal autofunnel. But you can get closer to the ideal: do not be afraid to carry out tests at every stage of the funnel. You can build a funnel by yourself, or you can keep your head free and entrust it to professionals. Like us:) (Originally published ) here

WTF Are Autofunnels and How To Design One for Your Business

React PWA with Cosmic JS In this tutorial I will show you how to build a progressive web app using React.js and . Our PWA will respond with status 200 when you are offline. The app will attain the last state of data and display it to you when you are offline. We will start with the basics and complete the app in full details. After completion we will deploy it to the Cosmic JS App Server, so let’s get started. Cosmic JS TL;DR Checkout the demo Download the codebase Admin Credentials: email: admin@abc.com password: admin What are We Building? As the title explained we are building a progressive web app (PWA) which will display news published by an admin. Let’s start by explaining what a PWA is in simple words: A PWA is a ordinary website that has some extra and unique features which make it extraordinary. Here are those features: You can use these apps while you are offline, that means these apps give a response code 200 while internet is disconnected Second most important feature is that you can install it on your device ( either desktop or mobile ) These apps are fully responsive in nature We will be implementing this app using React.js with the power of Cosmic JS. Let’s start. Requirements must be installed on your system to to run npm commands, with the help of which we will install react app. For more information, you can refer to . Node.js React Docs Getting Started Now that our system is configured with the required modules and tools, we are going to create a progressive web app project in React.js using Cosmic JS. Go to your any specific folder where you want to place your project and open console (terminal) and type the following command: npx create-react-app react-pwa here react-pwa is the name of your app and then goto created folder react-pwa and run command: npm start It will compile your basic React.js app and automatically opens in default browser. By default it runs on the port number 3000: http://localhost:3000/ Configure React App with Cosmic JS Till now, we have just implemented the front-end part of the PWA, now it is time to configure the server i.e. Cosmic JS server and database. It gives two in one facility. First of all, for free and create your first Bucket. Now create an Object-Type, which in our case is Admin. Similarly create two more object types named 1) News and 2) Comments. These three object types have different purposes: Admin will save the credentials of admin, News will save the news published by admin and Comments will save the comments posted by users. signup on Cosmic JS Now go to and generate read_key as well as write_key, and save all information of this page in a separate file on your system. Now place all of this information in the folder of your project and path will be: Settings > Basic Settings src>config>config.js To manage your project properly, you should create different folders for components, services, config files, css and images. It will help any third person to understand your code properly. All of these folders should reside in src folder. So create them first and jump to next step. Ready Made Bucket For this project you can directly import a bucket to your dashboard, in that bucket we have already created object-types. You have to download bucket.json from our and then just follow the steps: Choose default bucket > Go to Settings > go to Import / Export and click the button “Add import file”. It will import the bucket.json file. git repo Here we have created the platform to implement our PWA, so let’s move to the next one. Admin Login In our bucket, there is an object-type “Admin”. Create an object with title admin, and metafields email & password. You can give the values of email and passwords according to you. Here we gave email: and password: admin. Later you can use them for admin login in our demo. admin@abc.com Create a login page for admin in src>components>login.js On this page we are simply logging the admin in. We have statically created the email and password in the object-type “admin” and with those credentials we are logging him in. Here we imported the authentication service and in the onSubmit() function we are calling the login() function which is declared in the authentication service. In componentDidMount() we simply applied navigation when admin already logged in. Dashboard Component In this component we simply displayed the three links: Home, Add News and Logout. From here admin can navigate to main page, of can add a news or can logged himself out. On loading we applied navigation which will take the admin to home page if he is logged out and still trying to access this page. On logout() function we cleared the local storage and on the link Add News we navigated the admin to addnews.js. Location of this file is: src>components>dashboard.js Add News This component is accessible only for admin. He can post news over here. He can add title, description and body of the news. It will save the complete thread to object-type ‘news’ on the Cosmic JS server. Location of this component is: src>components>addnews.js Home Page On home page we are displaying all news which are published by the admin. First of all check its coding: This is the news.js and it is fetching as well as displaying the news on the home page of our app. We are explaining the working of getNews() method. If your device is online, then it is fetching data from backend and displaying on the template and also saving it on browser for offline use too. If the device is offline, then it will get the saved data from browser memory and will show to the user. Here you are seeing the title and description of the news and click to “Complete News” to read full story. This link will take you to next component: Single News Component This component is fetching the single news which you clicked from the home page. Check its coding first: It is displaying the single news as well as there is an option of comment also. Comments related to the post are displayed here and you can add comments also. singleNews() method is fetching the news at one time and we applied a filter to get the selected news. allComments() function is getting all comments and filtering the comments related to this single post only. Services There are two services named authentication service and news service. Let us explain them one by one: This is only for logging the admin in. There is one method defined which is searching the email and password supplied the the login page and returns the result to calling method. Authentication Service: This is the main service of this project which is handling almost everything here. Go through its coding first: News Service: Let’s explain its methods one by one: addNews(): It takes four parameters: title, description, newsbody and date and saves on the object type news getNews(): As the name specifies, it fetch all news from the object-type news submitComment(): This method submits the comment in the object type “comment” allComments(): This get the submitted comments in object-type “comments” on the Cosmic JS server Data Modelling We managed data on Cosmic JS properly, for this we created some object-types to understand the nature of each datum. Here is three object types : admin, news and comment. Admin stores the information of admin, and news stores the news published by admin. Comment object-type is storing the comments published by readers of our news app. What Have We Built Till Now? I want to make clear that our app is an ordinary app until now (in development mode); it doesn’t have any PWA features, so let’s start making it progressive in nature. When we created this app, it already created a file serviceWorker.js in the src folder, which handles the progressive nature of the app. At the time of online use, it automatically stores the static data of the app and reflects when the app is offline. Our app is not behaving a PWA because in serviceWorker it is coded to become a PWA only in production mode. So make its build and then run build, you will get some messages in the console like this: This is the static data capturing of the app. If you want to capture the dynamic data, you need to code it manually. We did this in the news.js file. We stored the app data when the app is online and it automatically called when app will be offline. So to enjoy its PWA feature, run the following commands: npm run build serve -s build Our app is partial-progressive in nature now. It is storing data while offline but it can’t be installed on the device yet. We’ll make that happen in the next step. Make App Complete PWA To make it complete PWA, here is the final step. We need to edit manifest.json, which is present in the public folder. It should look like: Here we added path of two icons, one for desktop and the other for mobile. Their size should be exactly the same and these two icons should be present in the public folder. Start_url is showing the path of build. This is the main basic coding of manifest and now our app is pure progressive in nature. In the above screenshot you can clearly check that Chrome browser is giving the option to install this React app on your system. Now we are checking out app with lighthouse tool, which is provided by Google to check PWAs, check output of lighthouse below: In report each milestone achieved by our app except 7th, which is an https related and will be fixed as we deployed it to our Cosmic JS server which supports HTTPs. Deploy on Cosmic Server Now our app is ready and needs some improvement before deployment on Cosmic server. Make sure the followings: A public GitHub repo ( private also available if you have connected your Git account ) should be matched Heroku requirements You need to follow some steps: Install express with command: npm install --save express Modify your package.json file, and add the following code Now create a file app.js in the root folder and add the following code in it Finally you made all necessary changes and it is now ready to deploy. Now go to: and click button deploy to Cosmic. After successful deployment you will receive an email and click on "view app" button to view the app. bucket>setting>deployment Verdict It is clear that with the help of Cosmic JS we can create powerful apps and can deploy on their server without any server setting or configuration. Here we created and deployed our React.js progressive application on Cosmic server easily without any difficulty. You don’t need to install any extra software to deploy over here. For any query or comment related to building apps in Cosmic JS, and . reach out to us on Twitter join the conversation on Slack

Stories, News & Mentions about USERPATH

Userpath’s Stories on HackerNoon