Corelight transforms network and cloud activity into evidence so that data-first defenders can stay ahead of ever-changing attacks. Delivered by our open NDR platform, Corelight’s comprehensive, correlated evidence gives you unparalleled visibility into your network. This evidence allows you to unlock new analytics, investigate faster, hunt like an expert, and even disrupt future attacks. Our on-prem and cloud sensors go anywhere to capture structured, industry-standard telemetry and insights that work with the tools and processes you already use. Corelight’s global customers include Fortune 500 companies, major government agencies, and research universities. Get started >> https://www.corelight.com/contact

Corelight

With the introduction of lockdown measures and brick-and-mortar stores closing their doors, retailers’ reliance on has intensified. The push to online and the need to attract and retain customers has made retail brands invest more in digital advertising and bring livestream shopping to their marketing mixes. ecommerce solutions A hot trend, livestream shopping makes online shopping more engaging and experiential while taking conversions way up. In this article, we’ll examine the phenomenon of livestream shopping, evaluate its impact on the retail sector, and zoom in on out-of-the-box and custom live video shopping tools. What is livestream shopping? The concept of live shopping goes back to the 80s and builds on the idea of video shopping. You are likely to remember QVC’s Lisa Robertson casually chatting with the viewers while demonstrating jewelry or HSN's Bobby Ray Carter showcasing fashion pieces with the viewers tuning in, seeing what’s for sale, and calling in to make a purchase. Livestream shopping — using video streaming tools to demonstrate products, communicate with an audience, and sell, all online — is a natural offshoot of this original TV-oriented model. Combining the best of ecommerce, video streaming, and social media, live commerce lets people shop anytime and from anywhere, encourages real-time communication, and sparks interactivity, appealing to today’s tech-savvy, sit-at-home customersLivestream shopping can take different forms, from live events and auctions to online marketplaces and influencer streams. Irrespective of the format, viewers can tune into a live shopping session, leave comments, ask questions, and communicate with a host or with one another. And once they are ready to purchase, they can click a link embedded in a shoppable video and proceed to checkout. Live shopping provides for personalization, too. Although one-to-many live streams, when products are marketed to mass online audiences, are more popular, brands can pay special treatment to their repeat buyers through one-to-one livestream sessions. With private dialogues between a host and a customer, brands can produce higher conversion rates and average order values and increase the customers’ chance of returning. How does livestream shopping work? Suppose you want to tap into livestream shopping. In that case, you have three options: 1. Leverage social media’s streaming functionality conditions and predicting patient outcomes 2. Market your products on an existing video commerce platform or a marketplace with streaming features 3. Opt for a custom livestream shopping solution 1. Social media The most popular and perhaps most straightforward way to start with livestream shopping is going live on popular social media apps. Luckily, most social media platforms have introduced livestream and ecommerce functionality in response to the video shopping trend. Instagram, for instance, has added built-in checkout capabilities throughout its apps, including Instagram Live, which, by the way, has seen a 70% increase in the number of viewers in 2020. Facebook, too, is developing native livestream shopping tools. In turn, TikTok is partnering with ecommerce giant Walmart for the latter to market their products online. The critical advantage of turning to social media is the possibility to team up with influencers, who play a crucial role in the success of livestream shopping, right where their audiences are. A study from Adweek reports that 49% of respondents actively follow social media influencers and based upon an influencer’s recommendation. So, having influencers as hosts may help in developing trust and loyalty among customers. However, as social media platforms generate income via media buy, they focus on being social. So, ecommerce functionality is still limited to select brands. As a result, shoppers have to be routed to other platforms or websites to finalize purchases, which disrupts the whole retailtainment experience. have made a purchase 2. Out-of-the-box live shopping apps Another way to start capitalizing on the video shopping trend involves joining an existing livestream shopping platform or an ecommerce marketplace with livestreaming features. And these span from video commerce giants like Amazon Live and Taobao, live shopping startups like Bambuser and Spin, and niche live commerce apps like NTWRK. The essential advantage is that most out-of-the-box video shopping apps feature built-in integrations with popular ecommerce platforms and payment gateways as compared to the previous option. So, users don’t have to be routed to other sites to complete a purchase and can enjoy an uninterrupted video shopping experience. The fundamental limitation is that some livestream shopping platforms, especially larger ones and those with established audiences, expect a brand to meet a range of requirements, like having a steady business history, and charge commissions. Amazon Live, for instance, charges up to 10% on each purchase. Another downside is rising competition — with millions of livestream views, it can be tricky for retailers to stand out. 3. Custom live video shopping applications Retailers who seek to make video shopping an essential part of their marketing kit may consider developing a custom retail solution with livestream functionality. Going custom is undoubtedly more effort-intensive than the previous two options. But a custom app can let retailers take maximum advantage of the video shopping trend with tailored functionality, high-quality streaming, advanced analytics, and maybe even some extra features, like AI-powered video content personalization. If you do opt for a we recommend starting with an MVP. However, it is essential to plan for the future and make sure your app will handle the growing number of shoppers. For that, make time to choose a reliable content delivery network (CDN). Looking for the perfect CDN, make sure it has a robust server infrastructure close to your target locations and pay attention to the bandwidth. With the basics in place, you can think about MVP features. The essentials to include in the first app version are: custom livestream shopping app, User registration and authorization Online streaming as such Live chatting Checkout and payment Server-side stream processing With the subsequent iterations, you can add everything from different signup options, personal customer profiles, search options, the possibility to save streams as on-demand videos, advanced analytics, streaming content personalization, and more What’s the livestream shopping market size? The current livestream shopping trend has its roots in China. China’s ecommerce giants started holding live shopping events hosted by local celebrities back in the 2010s. The US has begun to catch up with the trend, too. The US livestreaming market is expected to by the end of 2021 and hit $25 billion by 2023, according to Coresight Research. The market value is still incomparable to that in China, where it comprised $150 billion in 2020 and is expected to reach $300 billion in 2021. The companies that look prominent on the US livestream shopping market today include: reach $11 billion Launched in 2018, the platform has a 2-million follower base, which is constantly growing. According to Bloomberg, since its premiere, TalkShopLive’s sales have increased sevenfold TalkShopLive. . Having started as a directory for American retailers to reach Chinese travelers, the platform has turned into a profitable livestream shopping endeavor that has recently expanded into and is gaining popularity on the American market ShopShops . The mobile app uses live shows to sell limited edition products. The company has seen its sales double in 2020, with some shows reaching $1 million in sales in approximately 10 minutes NTWRK The ecommerce giant has rolled out its video streaming service with lifestyle, fitness, and cooking streams Amazon. The company has introduced Shoploop — a video shopping platform that features 90-second shoppable stories Google. Its Live Shopping feature allows creators and brands to sell products during Instagram Live broadcasts. The functionality is available to business accounts Instagram. An explosion of digital media and video content consumption, an increasing impact celebrities and influencers have on shopping decisions, and the push to online boosted by the COVID-19 pandemic allow projecting that more US retail brands will move on the live shopping trend in the near future. Companies profiting from livestream shopping today Nordstrom Nordstrom, an American department store chain, launched its own in 2020. Following the launch, the brand hosted over 50 virtual events, translating the store experience into the digital world, and plans on carrying out their livestream shopping endeavors through 2021. During each event, customers can shop the fashion products mentioned in the video and participate in a live chat. The company’s efforts are paying off. In 2020, 54% of the brand’s sales came from digital channels, while virtual styling accounted for 30% of all styling appointments. shoppable livestream channel CAIA Cosmetics, Lyko CAIA Cosmetics and Lyko, Swedish cosmetic and hair care brands, have partnered with Bambuser, a video commerce startup, to power their marketing with live shopping. CAIA Cosmetics, for instance, holds make-up tutorials with the brand’s founder Bianca Ingrosso and claims to be very satisfied with the customers’ response. The , with 17% of the viewers participating in the live chat and 60% — engaging through likes. The first live stream ended up with a 5% conversion rate, which is higher than the brand’s website conversions. Lyko, in turn, chose to kick off their live broadcasts by offering tips and gifts during the holiday season. The brand had an average viewing time of 16.5 minutes, 17% chatting and 29% liking engagement, and a 22% conversion rate. average stream viewing time comprises 11 minutes Quivr Quivr, a coffee and tea manufacturer, has tried social media to market their products via video streams but eventually landed on Amazon Live as their video commerce platform. Quivr’s co-founder Ash Crawford says that the of about 150% the next 24 hours after a live stream. brand sees a guaranteed increase in sales Livestream shopping trends and predictions The possibilities for further evolution and expansion of livestream shopping are numerous. Introducing AI, for example, can help retailers create highly personalized channels of live, interactive content created or curated specifically for a given customer. and VR, in turn, would be of help in creating immersive video shopping experiences for sit-at-home customers, for example, transporting them to the front row of a fashion show with a host ready to answer questions. Leveraging AR In the shorter perspective, brands will continue to extend their reach to more customers through live streams, analysts say. Considering the drastic change in shopping habits and the push to online spaces that are expected to last, livestream shopping is likely to extend in the post-pandemic world. Livestream shopping offers an opportunity to bring the best of physical shopping to online customers. If you’re ready to seize the opportunity, contact the ITRex team!

An Introduction to Livestream Shopping

Beating Amazon with Blockchain

Content Delivery Networks (CDNs) are commonly used with AEM to improve website performance by and from edge locations closer to users. However, when AEM content changes it can lead to in the . Manual cache invalidation is complex and error-prone. caching serving content outdated copies CDN cache Though CDNs improved performance, our authors’ new changes weren’t reflected until caches expired. post-publish did not scale. Clearing the cache manually I could not find an existing out-of-the-box solution for this problem. So I developed an that and selective cache based on . open-source AEM library listens to publishing events automatically triggers invalidation configurable rules The library AEM replication events or resource changes and then runs to invalidate cached CDN content related to the published updates. allow various CDNs while enable customization. reacts to asynchronous background jobs Flexible APIs integration with extension points The Problem CDNs cache website content like pages, images, and JS/CSS assets. This avoids repeatedly serving from the AEM publish tier which . The CDN only re-fetches assets when they from the after a . static content improves response times expire cache TTL (Time to Live) However, updated AEM content can get lingering in the CDN after publishing. Some examples: outdated copies a new content page is not immediately to users. The older version remains cached. Publishing visible a product image in DAM does not show until the CDN cache expires. Updating Depending on the business context, any of these situations can be a really that needs to be addressed urgently. But sadly, on every publish is and : sensitive issue clearing the cache manually operationally complex time-consuming Identifying related URLs is manual and repetitive Tedious cache tagging schemes to coordinate flushing An automated solution is needed… one that selectively and intelligently invalidates cache only for published updates. Automated Cache Invalidation Library “Automatic CDN Cache Invalidator for AEM” listens to publish events and resource changes, then triggers background cache invalidation per configurable rules. Flexible APIs allow integration with various CDNs. https://github.com/realgpp/aem-cdn-cache-invalidator?embedable=true Key Capabilities : Reacts to AEM replication events for publishing updates Automatic triggering Invalidates cache only for specific published paths rather than mass flush Selective invalidation: Uses queue-based jobs framework to offload CDN calls Async processing: Flush cache by URL, tag or custom criteria Configurable rules: Detailed logging for tracking Logs: Akamai Purge API is integrated Akamai support built-in: Easily extendable for other CDNs via APIs Flexible CDN integrations: Hooks to personalize behavior Extension APIs: It solves manual cache management challenges allowing authors to focus on content while optimizing end-user experience. Let’s walk through the flow… How it Works The application has configurable OSGi services and listeners for cache invalidation triggered by AEM events. : Replication or resource change events signify content updates. Event listening An OSGi listener consumes events filtering to specific trees. It creates asynchronous jobs with updated paths. Selective listener: A separate job thread processes paths to extract cache-related metadata like URLs, tags etc. Background job: The job makes the API call to invalidate the cache for the computed metadata. CDN API call: The CDN receives the API request and flushes related cached content. Cache flush: The async approach avoids publish request performance impact. The CDN calls happen out-of-band without blocking AEM. The entire flow is configurable via OSGi including supported events, cache criteria, URLs, or logic. Out-of-the-box integrations make it easy to get started while extension points allow customization. Let’s explore common configuration options… Purging content with Akamai API The project includes reusable services, listeners, and jobs supporting caching scenarios like: Invalidating pages or assets by URL Website sections by or tag code The following are the Akamai Purge APIs supported out of the box: and by Tag invalidation deletion and by CP Code invalidation deletion and by URL invalidation deletion The following block, which you can find in the ui.config.example maven module, shows how to configure the Akamai service that comes with the library. // file com.baglio.autocdninvalidator.core.service.impl.AkamaiInvalidationServiceImpl.cfg.json
{
  "isEnabled": true,
  "configurationID": "cdn-akamai",
  "hostname": "localhost:8443",
  "getAkamaiClientToken": "placeholder_akamai_client_token",
  "getAkamaiAccessToken": "placeholder_akamai_access_token",
  "getAkamaiClientSecret": "placeholder_akamai_client_secret",
  "httpClientConfigurationID": "http-client-akamai",
  "purgeType": "invalidate"
} This service lets you choose how to purge content from the CDN: you can either invalidate or delete it. As you can see, you need to replace placeholders with your Akamai values. You also need a dedicated HTTP Client Configuration to work with the Akamai APIs. After that, your service can be used by other sling jobs that handle content-change events. Now let’s explore other configuration options that let you customize the library without coding… Configuration Areas The library offers a variety of configuration options to customize its functionality without code customizations. You can adjust these settings to suit your specific needs and preferences. I want to keep this introduction brief and engaging but don’t worry if you have questions about the details. You can find more information in the README.md file of the project or in the Javadoc comments if you like to explore the code. Key aspects to configure are: Replication topics or resource change events that trigger the invalidation jobs. Observed events: // file com.baglio.autocdninvalidator.core.listeners.ReplicationEventListener-website-generic.cfg.json
{
  "isEnabled": true,
  "resource.paths": [
    "/content/we-retail",
    "/content/wknd"
  ],
  "job.topic": "com/baglio/autocdninvalidator/akamai/generic",
  "filter.regex": "^/content/(we-retail|wknd)(?!.*products)(?!.*adventures).*$"
} // file com.baglio.autocdninvalidator.core.listeners.DynamicResourceChangeListener-website-generic.cfg.json
{
  "isEnabled": false,
  "resource.change.types": [
    "CHANGED"
  ],
  "resource.paths": [
    "/content/we-retail",
    "/content/wknd"
  ],
  "job.topic": "com/baglio/autocdninvalidator/akamai/generic",
  "filter.regex": "^/content/(we-retail|wknd)(?!.*products)(?!.*adventures).*$"
} API credentials and settings for your provider. CDN credentials: // file com.baglio.autocdninvalidator.core.service.impl.AkamaiInvalidationServiceImpl.cfg.json
{
  "isEnabled": true,
  "configurationID": "cdn-akamai",
  "getAkamaiClientToken": "placeholder_akamai_client_token",
  "hostname": "localhost:8443",
  "getAkamaiAccessToken": "placeholder_akamai_access_token",
  "getAkamaiClientSecret": "placeholder_akamai_client_secret",
  "httpClientConfigurationID": "http-client-akamai"
} Filtering rules and cache flush criteria for each job consumer. Invalidation rules: // file com.baglio.autocdninvalidator.core.jobs.EditorialAssetInvalidationJobConsumer.cfg.json
{
  "isEnabled": true,
  "job.topics": [
    "com/baglio/autocdninvalidator/akamai/generic"
  ],
  "cdnConfigurationID": "cdn-akamai",
  "invalidation.type": "tag",
  "tagCodeMappings": [
    "/content/we-retail/(..)(/.*)*=tag-dev-$1",
    "/content/we-retail/(..)/.*/experience/.*=tag-dev-$1-experience",
    "/content/wknd/(..)(/.*)*=tag-dev-$1",
    "/content/wknd/(..)/.*/adventures/.*=tag-dev-$1-adventures"
  ]
} Throttling and parallelization configs for each job topic. Please have a look at the section Custom Job Queue Configuration of project README file. Job topic tuning: These configurations determine how the library responds to AEM events, how many cache jobs run concurrently, and how the service communicates with your CDN provider to purge the cache. Now that we have learned the basics, let’s move on to the fun part where we show you how to make the invalidation work on the AEM instance… Enabling Akamai CDN Integration If you want to use this library right away, you don’t need to write any code. Just follow these simple steps: : Install the ui.config.example module or create the manually. The ui.config.example module is a sample configuration that provides a working example for . You can install it from the package manager or create the OSGi config manually through the web console. Step 1 OSGi config Akamai integration : Enter your Akamai and in the . You must provide your Akamai hostname, client token, access token, and client secret in the AkamaiInvalidationServiceImpl config file. You can find these values in your Akamai portal. Step 2 credentials settings AkamaiInvalidationServiceImpl config file : Choose to AEM events on Author or Publish.Depending on your setup, you can choose to listen for AEM events on Author or Publish. If you choose Author, the module will detect and the accordingly. If you choose Publish, the module will detect and the accordingly. Step 3 listen for publish events invalidate cache content updates purge cache Now let’s walk through adding support for a new custom CDN provider… Implementing a Custom CDN Provider Now that we’ve explored utilizing the library without coding, what if you encounter the need for a CDN provider that isn’t currently supported? The library is designed for to support new CDN providers. You can easily extend it by following these steps: extension Create a Java class the . This interface defines the methods for the . implementing CdnInvalidationService interface purging CDN cache Implement the logic for your CDN’s in your implementation class. calling purge API Create an for your class with the necessary and property values. Make sure to give it a so that it can be identified by the invalidation job. OSGi configuration credentials unique configuration ID Use the configuration ID mentioned above when you create configurations. invalidation job The library provides the invalidation logic when using the built-in Akamai integration. If you need to the or , the class exposes you can without having to reimplement the full Akamai handling. flexibility to customize tweak preprocessing postprocessing AbstractInvalidationJob hook methods override With the procedure shown in the above image, you get the benefits of leveraging the existing Akamai capabilities while adapting things like: - The and methods allow transforming the content paths and invalidation values before and after generating them. preprocessInvalidationValues postprocessInvalidationValues - The and methods allow updating the public URLs before and after resolving them. preprocessPublicUrls postprocessPublicUrls - The and methods enable logic before and after making the CDN purge call. beforeInvalidation afterInvalidation By selectively overriding these hooks, you can adapt the data and flows to your needs while reusing the bulk of the Akamai integration code. This makes the library very extensible without having to start from scratch. Wrap Up And that’s a wrap! We’ve covered everything from to with this CDN invalidator library. I hope you found it an interesting read! out-of-the-box setup fully custom integrations Did any part leave you scratching your head? Or maybe this library has already helped you invalidate caches more easily? Either way, I’d love to hear your thoughts! Let me know what worked, what didn’t, or what other guides would help improve your CDN workflows. Even better — I welcome if you have , tweaks, or new snippets that can this library and tutorial! pull requests code changes enhance I greatly appreciate you taking the time to provide and . Your will help this project and craft content that empowers fellow developers to their . feedback contributions real-world experiences evolve master CDN footprints Also published . here

How to Automate CDN Cache Invalidation in Adobe Experience Manager

Music often plays a significant role in video games. It can create an immersive atmosphere, convey the game's tone, and even affect gameplay. With that in mind, it's no surprise that some of the best video game music that comes out each year can be found in both triple-A and indie titles. But what exactly makes for an excellent game soundtrack? Is it the catchiness of the melody? The way it compliments the game's visuals? Or simply the fact that it gets stuck in your head for days on end? There's no one answer, but we can narrow it down to a few key characteristics. The Best Video Game Music of 2022 First and foremost, excellent video game music is memorable. It's the kind of music you can't help but hum or sing along to, even when you're not playing the game. It also needs to be well-suited to the game it's accompanying. A great soundtrack enhances the experience of playing by setting the perfect mood and atmosphere. And finally, it should be something you want to listen to outside of the game. With all that in mind, here are some of the 2022 games with the best music. Horizon Forbidden West Stray OlliOlli World Teenage Mutant Ninja: Shredder's Revenge Elden Ring Halo Infinite Metal: Hellsinger Cult of the Lamb 1. Horizon Forbidden West : The Flight, Oleksa Lozowchuk, Joris de Man, Niels van der Leest Composers Horizon Forbidden West is a sequel to 2017's Horizon Zero Dawn. It is an open-world action RPG set in a post-apocalyptic world overrun by animalistic machines. The game picks up where the first one left off, with Aloy, a skilled Nora hunter, leading a group of companions on an expedition to the Forbidden West, searching for answers about a deadly plague. The game's score is just as sweeping and majestic as the first, with an added sense of urgency and excitement. It is also a perfect example of video game music that fills you with excitement and desire to play more and more. 2. Stray : Yann Van Der Cruyssen Composers Stray is a third-person stealth action game developed by BlueTwelve Studio and one of the best sci-fi games of 2022. Set in a future Tokyo, the game allows you to take control of a stray cat, who, after falling into a walled city full of robots, machines, and mutant bacteria, must return to the surface with the help of a drone companion. Stray is a unique and atmospheric game whose music flawlessly complements the ethereal atmosphere. Characterized by an ambient, dreamlike quality, the game's music is both beautiful and terrifying. 3. OlliOlli World : various artists Composers OlliOlli World is a charming and colorful skateboarding game with an equally vibrant soundtrack. The game's music is one of its standout features, and it's hard to imagine the game without it. With a mix of upbeat electronic tunes and more mellow, reflective tracks, the game's music perfectly sets the tone of the skateboarding adventure, and no doubt deserves a playlist of its own in your music library. 4. Teenage Mutant Ninja: Shredder's Revenge : Tee Lopes Composers If you're of a certain age, the Teenage Mutant Ninja Turtles were a big part of your childhood with its music. The 1990s film is still fondly remembered by many, and the recent resurgence in popularity of the franchise has been accompanied by a new game, Teenage Mutant Ninja: Shredder's Revenge. The side-scrolling beat ‘em up game is a love letter to the classic TMNT games of the '80s and '90s. And the game's score perfectly captures that feeling, with high-energy tracks that will get your blood pumping. There is no doubt that TMNT: Shredder's Revenge boasts some of the best video game music of the year. 5. Elden Ring : Yuka Kitamura, Tsukasa Saitoh, Shoi Miyazawa, Yoshimi Kudo, Tai Tomisawa Composers Many consider Elden Ring the best game of 2022, a dark and atmospheric action RPG set in a world of gods and mortals. The game follows the story of a character on a journey to repair the Elden Ring and become the new Elden Lord. The music of the game perfectly matches its dark and brooding atmosphere. The game’s score is beautiful and inspiring and features a mix of orchestral and choral tracks that will send chills down your spine on more than one occasion. 6. Halo Infinite : This Will Destroy You, Joel Corelitz, Gareth Coker, Curtis Schweitzer Composers Halo Infinite is the long-awaited return of one of gaming's most iconic franchises, and its music does not disappoint. The game's score is nostalgic and fresh, with new takes on classic Halo themes. And it's unlikely that you will find a better opening score than Zeta Halo. 7. Metal: Hellsinger : Two Feathers Composers No video game music roundup would be complete without mentioning Metal: Hellsinger, the heavy-metal-themed rhythm game. The music is, of course, metal as hell and perfectly complements the game's fast-paced gameplay. The game's loud and aggressive music will get your head banging for a long time after you put down the controller. 8. Cult of the Lamb : Narayana Johnson Composers Cult of the Lamb is a dark and atmospheric game about a religious cult. Despite the eerie topic, the game's music is not too spooky. It features many catchy and upbeat tracks and atmospheric scores that keep you focused on the task, making it great music for coding or similar activities. Final Words on the Best Video Game Music of the Year So there you have it, some of the best video game music of the year. The year is not over, so maybe we will see some more excellent game soundtracks before it's done. But for now, these are my favorites. What are yours? Let me know in the comments.

The Best Video Game Music of 2022

La mejor música de videojuegos de 2022

La meilleure musique de jeu vidéo de 2022

2022 का सर्वश्रेष्ठ वीडियो गेम संगीत

2022 年のベスト ビデオ ゲーム音楽

A melhor música de videogame de 2022

Nhạc trò chơi điện tử hay nhất năm 2022

2022 年最佳视频游戏音乐

The Evolution of Modern Medicine: Chapter V — THE RISE AND DEVELOPMENT OF MODERN MEDICINE

Scraping websites built for modern browsers is far more challenging than it was a decade ago. jsoup is a convenient API that makes scraping websites trivial via DOM traversal, CSS Selectors, JQuery-Like methods and more. But it isn’t without its caveat. Every scraping API is a ticking time bomb. Real-world HTML is flaky. It changes without notice since it isn’t a documented API. When our Java program fails in scraping, we’re suddenly stuck with a ticking time bomb. In some cases, this is a simple issue that we can reproduce locally and deploy. But some nuanced changes in the DOM tree might be harder to observe in a local test case. In those cases, we need to understand the problem in the parse tree before pushing an update. Otherwise, we might have a broken product in production. What is jsoup? The Java HTML Parser Before we go into the nuts and bolts of debugging jsoup let’s first answer, the question above and discuss the core concepts behind jsoup. The defines it as: jsoup website jsoup is a Java library for working with real-world HTML. It provides a very convenient API for fetching URLs and extracting and manipulating data, using the best of HTML5 DOM methods and CSS selectors. jsoup implements the specification and parses HTML to the same DOM as modern browsers do. WHATWG HTML5 With that in mind, let’s go directly to a simple sample also from the same website: Document doc = Jsoup.connect("https://en.wikipedia.org/").get();
log(doc.title());
Elements newsHeadlines = doc.select("#mp-itn b a");
for (Element headline : newsHeadlines) {
  log("%s\n\t%s",
    headline.attr("title"), headline.absUrl("href"));
} This code snippet fetches headlines from wikipedia. In the code above, you can see several interesting features: Connection to URL is practically seamless – just pass a string URL to the connect method There are special cases for some element children. E.g. Title is exposed as a simple method that returns a string without selecting from the DOM tree However, we can select the entry using pretty elaborate selector syntax If you’re looking at that and thinking “that looks fragile”. Yes, it is. Simple jsoup Test To demonstrate debugging, I created a simple demo that you can download here. You can use the following Maven dependency to install jsoup into any Java program. Maven will download jsoup jar seamlessly: <dependency>
  <groupId>org.jsoup</groupId>
  <artifactId>jsoup</artifactId>
  <version>1.14.3</version>
</dependency> This demo is a trivial Java app that returns a complete list of external links and elements with src attributes in a page. This is based on the code from here, converted to a Spring Boot Java program. The jsoup applicable code is relatively short: public Set<String> listLinks(String url, boolean includeMedia) throws IOException {
   Document doc = Jsoup.connect(url).get();
   Elements links = doc.select("a[href]");
   Elements imports = doc.select("link[href]");

   Set<String> result = new TreeSet<>(String.CASE_INSENSITIVE_ORDER);
   if(includeMedia) {
       Elements media = doc.select("[src]");
       for (Element src : media) {
           result.add(src.absUrl("src"));
           //result.add(src.attr("abs:src"));
       }
   }

   for (Element link : imports) {
       result.add(link.absUrl("abs:href"));
   }

   for (Element link : links) {
       result.add(link.absUrl("abs:href"));
   }

   return result;
} As you can see, we fetch the input String URL. We can also use input streams, but this makes things slightly more complicated when parsing relative URLs (we need a base URL anyway). We then search for links and objects that have an src attribute. The code then adds all of them into a set to keep the entries sorted and unique. We expose this as a web service using the following code: @RestController
public class ParseLinksWS {
   private final ParseLinks parseLinks;

   public ParseLinksWS(ParseLinks parseLinks) {
       this.parseLinks = parseLinks;
   }

   @GetMapping("/parseLinks")
   public Set<String> listLinks(@RequestParam String url, @RequestParam(required = false) Boolean includeMedia) throws IOException {
       return parseLinks.listLinks(url, includeMedia == null ? true : includeMedia);
   }
} Once we run the application can the application, we can use it with a simple curl command: curl -H "Content-Type: application/json" "http://localhost:8080/parseLinks?url=https%3A%2F%2Flightrun.com" This prints out the list of URLs referred to on the Lightrun home page. Debugging Content Failures Typical string scraping issues occur when an element object changes. E.g. Wikipedia can change the structure of its pages and the select method above can suddenly fail. This is often a nuanced failure, e.g. missing DOM element in the Java object hierarchy which can trigger a failure of the select method. Unfortunately, this can be a subtle failure. Especially when dealing with nested node elements and inter-document dependencies. Most developers solve this by logging a huge amount of data. This can be a problem due to two big reasons: Huge logs – they are both hard to read and very expensive to ingest Privacy/GDPR Violations – a scraped site might include user specific private information. Worse! The scraped site might change to include private information after scraping was initially implemented. Logging this private information might violate various laws. If we don’t log enough and can’t reproduce the issue locally, things can become difficult. We’re stuck in the add logs, build, test, deploy, reproduce – rinse repeat loop. Lightrun offers a better way. Just track the specific failure directly in production, verify the problem, and create a fix that will work with one deployment. This tutorial assumes you installed Lightrun and understand the basic concepts behind it. If not, please . NOTE: check out the docs Finding your way in Browser DOM Assuming you don’t know where to look, a good place to start is inside the jsoup API. This can lead you back to user code. The cool thing is that this works regardless of your code. We can find the right line/file for the snapshot by digging into the API call. I ctrl-clicked (on Mac use Meta-click) the select method call here: Elements links = doc.select("a[href]"); And it led me to the Element class. In it I ctrl-clicked the Selector “select” method and got to the “interesting” place. Here, I could place a conditional snapshot to see every case where an “a[href]” query is made: This can show me the methods/lines that perform that query: This can help a lot in narrowing down the general problematic area in the document object hierarchy. Sometimes, a snapshot might not be enough. We might need to use a log. The advantage of logging is that we can produce a lot of information, but only for a specific case and on-demand. The value of logs is that they can follow an issue in a way that’s very similar to stepping over code. The point where we placed the snapshot is problematic for logs. We know the query sent but we don’t have the value that’s returned yet. We can solve this easily with logs. First, we add a log with the following text: "Executing query {query}" Then, to find out how many entries we returned, we just go to the caller (which we know thanks to the stack in the snapshot) and add the following log there: Links query returned {links.size()} This produces the following log which lets us see that we had 147 links. The beauty of this is that the additional logs are interlaced with the pre-existing logs in-context: a[href] Feb 02, 2022 11:25:27 AM org.jsoup.select.Selector select
INFO: LOGPOINT: Executing query a[href]
Feb 02, 2022 11:25:27 AM com.lightrun.demo.jsoupdemo.service.ParseLinks listLinks
INFO: LOGPOINT: Links query returned 147
Feb 02, 2022 11:25:27 AM org.jsoup.select.Selector select
INFO: LOGPOINT: Executing query link[href]
Feb 02, 2022 11:25:27 AM org.jsoup.select.Selector select
INFO: LOGPOINT: Executing query [src] Avoid Security and GDPR Issues GDPR and security issues can be a problem with leaking user information into the logs. This can be a major problem, and Lightrun helps you reduce that risk significantly. Lightrun offers two potential solutions that can be used in tandem when applicable. Log Piping The big problem with GDPR is the log ingestion. If you log private user data and then send it to the cloud, it’s there for a long time. It’s hard to find after the fact and it’s very hard to fix. Lightrun provides the ability to pipe all of Lightrun’s injected logging to the IDE directly. This has an advantage of removing noise from other developers who might work with the logs. It can also skip the ingestion (optionally). To send logs only to the plugin, select the piping mode as “plugin”. PII Reduction/Blocklists Personally Identifiable Information (PII) is at the core of GDPR and is also a major security risk. A malicious developer in your organization might want to use Lightrun to siphon user information. Blocklists prevent developers from placing actions in specific files. PII reduction lets us hide information matching specific patterns from the logs (e.g. credit card format etc). This can be defined in the Lightrun web interface by a manager role. TL;DR With Java content scraping, jsoup is the obvious leader. Development with jsoup is far more than string operations or even handling the connection aspects. Besides getting the document object, it also handles complex aspects required for DOM element and scripting. Scraping is a risky business. It might break in the blink of an eye when a website changes slightly. Worse, it can break to some users in odd ways that are impossible to reproduce locally. Thanks to Lightrun, we can debug such failures directly in the production environment and publish a working version swiftly. You can use Lightrun for free by signing up . here Also Published Here

Debugging jsoup Java Code in Production

Introduction In this blog, we have detailed the approach of how to use Spark on Kubernetes and also a brief comparison between various cluster managers available for Spark. Spark Spark is a general-purpose distributed data processing engine designed for fast computation. The main feature of Spark is its in-memory cluster computing that increases the processing speed of an application. It supports workloads such as batch applications, iterative algorithms, interactive queries and streaming. During execution, it creates the following components: Driver Executor To manage these components, there is a cluster manager that takes care of resource allocation. Following are the various options available for cluster manager: Standalone Apache Mesos Hadoop YARN Kubernetes Standalone Cluster Manager To use Spark Standalone Cluster manager and execute code, there is no default high availability mode available, so we need additional components like Zookeeper installed and configured. Hadoop YARN YARN (Yet Another Resource Negotiator) focuses on distributing MapReduce workloads and is majorly used for Spark workloads. Apache Mesos The Mesos kernel runs on every machine and provides applications with APIs for resource management, scheduling across the entire datacenter, and cloud environments. It provides a cluster manager which can execute the Spark code. Kubernetes It uses the kube-api server as a cluster manager and handles execution. Comparison between Hadoop YARN and Kubernetes – as a cluster manager Kubernetes is a container orchestration engine which ensures there is always a high availability of resources. Apart from that it also has below features. Brief introduction Kubernetes and its components Self healing Automatic rolling updates and rollback Resource management Service discovery Load balancing Service discovery Architecture of Kubernetes The architecture of Kubernetes has 2 major components, they are: 1. Master components All the requests from the user using API, kubectl are sent to master component that is the API Server converts json or yaml requests to http call API Server contains the details of the cluster and its components and current state ETCD ensures that the cluster is always in the desired state Controller takes care of object creation based on resource availability Scheduler 2. Node components The following are the components that are come under the nodes: Kubelet is the agent which takes care of the execution of the tasks which have been assigned to it and reports back the status to the api server Kube Proxy is the networking component that takes care of networking related tasks Container Runtime it provides an environment on the nodes for container execution Other components Kubectl: a utility used to communicate with the Kubernetes cluster Spark execution on Kubernetes Below is the pictorial representation of spark-submit to API server. We can use spark-submit directly to submit a Spark application to a Kubernetes cluster. Once submitted, the following events occur: Creation of a Spark driver running as a . Kubernetes pod Creation of executors which are also run within Kubernetes pods, connects to them, and executes the application code Termination and clean up of executor pods occur when the application completes. But the driver pod persists, logs and remains in “completed” state in the Kubernetes API until it’s eventually garbage collected or manually cleaned up. There are 2 options available for executing Spark on a EKS cluster: Option 1: Using Kubernetes Master as Scheduler Option 2: Using Spark Operator Option 1: Using Kubernetes master as scheduler Below are the prerequisites for executing spark-submit using: A) Docker image with code for execution B) Service account with access for creation of pods, services, secrets C) Spark-submit binary in local machine A. Creating Docker image for Java and PySpark execution Download Spark binary in the local machine using this link https://archive.apache.org/dist/spark/ In this path spark/kubernetes/dockerfiles/spark there is Dockerfile which can be used to build a docker image for jar execution Ensure that you are in the Spark directory as it needs jars and other binaries to be copied. So it uses all the directories as context. docker build -t latest -f kubernetes spark/Dockerfile . spark: /dockerfiles/ Once this image is built, it can be used as a base image for the other code execution. Docker image creation for PySpark code execution: In this path: there is a ready Docker file which will be used for PySpark execution. spark/kubernetes/dockerfiles/spark/bindings/python Ensure that you are in the Spark directory as it needs jars and other binaries to be copied so it will use all the directories as context. docker build -t latest -f \
kubernetes spark dockerfiles bindings Dockerfile . sparkpy: /dockerfiles/ /kubernetes/ /spark/ /python/ B. Creating Kubernetes service account and cluster-role binding Components that will be used: Service account: an account which will be used for authentication of processes running inside the pods Cluster-Role: defines the access for the service account across the cluster Cluster-Role Binding: binds and creates the role with the service account Create a yaml file with below contents apiVersion: v1 kind: ServiceAccount metadata: name: spark-sa --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: namespace: default name: spark-role rules: - apiGroups: [""] resources: ["pods", "services" , "configmaps" ] verbs: ["create", "get" , "watch" , "list" , "post" , "delete" ] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: spark-role-binding subjects: - kind: ServiceAccount name: spark-sa roleRef: kind: ClusterRole name: spark-role apiGroup: rbac.authorization.k8s.io kubectl apply -f rbac.yaml Getting cluster information: cluster- kubectl info This command gives the master url it and it will look as shown below Kubernetes master is running at https: //ABCDZZZZZZZZZZZZZZZ.sk1.region.eks.amazonaws.com Make a note of this url C. Executing Spark-submit: Now go to the directory which has Spark binary and use the below command: ./spark-submit  
--deploy-mode cluster
--master <master_url> 
--conf spark =spark-sa
--name spark-pi
--conf spark = --conf spark =<docker image with code>  
--conf spark =<docker image with code>  
local: .kubernetes .authenticate .driver .serviceAccountName .executor .instances 5 .kubernetes .driver .container .image .kubernetes .executor .container .image ///opt/spark/examples/src/main/python/pi.py The driver and executor will be created with whatever name is specified as app name. Using the driver pod we can view logs, access url below commands can be used for it. To view logs kubectl logs -f spark- - - pi 1581014997671 driver To view spark ui kubectl port-forward (driver-pod-name) : 4040 4040 Post execution, the drivers will be available based on the execution state, until it is deleted manually. Additional useful options that can be used with Spark-submit Environment Variables: When the script requires any environment variable that need to be passed, it can be done using Kubernetes secret and referred to it. Details of achieving this are given below. “ ”: “ ”, spark .kubernetes .executor .secretKeyRef .DB_PASS snowsec :db_pass Create a secret with the name snowsec and in that db_pass is the key and which will be referred to the spark environment using DB_PASS. Adding labels to the pod: When we want to add additional labels to pod we can use below options For driver pod: spark.kubernetes.driver.label.[LabelName] For executor pod: spark.kubernetes.executor.label.[LabelName] Using node affinity: We can control the scheduling of pods on nodes using selector for which options are available in Spark that is . ​ spark .kubernetes .node .selector [labelKey] Option 2: Using Spark operator on Kubernetes Operators Operator is a method of packaging, deploying and managing a Kubernetes application. Kubernetes application is one that is both deployed on Kubernetes, managed using the Kubernetes APIs and kubectl tooling. Using Spark operator on Kubernetes Official link https://operatorhub.io/operator/spark-gcp Installing Kubernetes operator on EKS: Prerequisites: Helm needs to be installed and configured Verify Helm installation using below command helm version –short Adding Helm repo to running EKS Helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator Installing Helm repo on EKS : helm install incubator/sparkoperator 
--name sparkv24 
--namespace -- =default  
-- = default set sparkJobNamespace set enableWebhook true Verifying Installation: helm ls Once applied, the following components will be created: Spark-operator-deployment Spark-serviceaccount Spark-rbac Spark-operator-serviceaccount Spark-operator-rbac Webhook-init-job Webhook-service Here we need the job in yaml. Once we have yaml file, we can submit the job using the below command: kubectl -f apply The Yaml file looks as follows: : : SparkApplication : : spark-pi-jar : default : : Scala : cluster : : Always : org.apache.spark.examples.SparkPi : : : : Never :
    - : : : : Directory : : : : : : . : sparkv24-spark :
      - : : : : : : : : . :
      - : : apiVersion "sparkoperator.k8s.io/v1beta2" kind metadata name namespace spec type mode image "gcr.io/spark-operator/spark:v2.4.4" imagePullPolicy mainClass mainApplicationFile "local:///opt/spark/examples/jars/spark-examples_2.11-2.4.4.jar" sparkVersion "2.4.4" restartPolicy type volumes name "test-volume" hostPath path "/tmp" type driver cores 1 coreLimit "1200m" memory "512m" labels version 2.4 4 serviceAccount volumeMounts name "test-volume" mountPath "/tmp" executor cores 1 instances 1 memory "512m" labels version 2.4 4 volumeMounts name "test-volume" mountPath "/tmp" Once we submit the job, it will create 2 pods: Executor Driver We can verify these using kubectl pods get Logs by using below command logs -f pod_name kubectl Details of submitted job using below command kubectl describe sparkapplication Contains details about Web-UI, service and events that occurred during creation. Access Web UI: To access the Web UI for a long running job can be done using port forwarding, using the below mentioned command kubectl port-forward (driver-pod-name) : 4040 4040 Conclusion Key considerations for Production Spark code on Kubernetes 1. Cost-Effective No requirement of up and running infrastructure to use Spark on EKS. 2. Build and Deployments As we deploy the Docker image with the Spark submit, so when we have code changes we need to pass the docker image with Spark submit. These images can be tagged to track the changes. 3. Ideal Use Cases When workload is less (e.g. 8-10 hr job executions per day) and as batch processing. 4. Availability/Fault Tolerance Kubernetes has the scheduler which manages the pods created as driver and executor. This enables the usage of pods based on resource availability. Quotas for a namespace can be assigned for better resource management. 5. Resource Tracking We have used node selectors in Spark submit which allows us to run specific workloads on a specific node. This in turn allows us to track the usage of resources. 6. Monitoring We have integrated Spark workloads monitoring with Prometheus and Grafana, by using Kube-state-metrics and creating a dashboard. We do a Spark submit by assigning pod labels which allow us to create custom dashboards for specific labels. 7. Logging We have used the ELK stack for visualizing the logs. The approach we have detailed is suitable for pipelines which use spark as a containerized service. It also ensures optimal utilization of all the resources as there is no requirement for any component, up and running before doing Spark-submit. Additionally, Spark can utilize features like namespace, quotas along with other features of Kubernetes. About the Author Ajaykumar Baljoshi is a Senior Devops Engineer at , who currently works on Containerization, Kubernetes, DevOps and Infrastructure as Code. Sigmoid Previously published at https://www.sigmoid.com/blogs/containerization-of-pyspark-using-kubernetes/

CORELIGHT

corelight.com

201-500 employees

Since 2013

COMPANY RANKING

CORELIGHT

EVERGREEN INDEX #3457

Representing this company? Click here to claim and customize this page!

Corelight WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

CORELIGHT

corelight.com

201-500 employees

Since 2013

COMPANY RANKING

CORELIGHT

EVERGREEN INDEX #3457

RELATED COMPANIESRANK

STORIES ABOUT CORELIGHT

Representing this company? Click here to claim and customize this page!

Corelight WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!