The Shelf

theshelf

Real business problems require serious custom solutions at a reasonable price. Businesses that invest in pertinent technology to make informed decisions and automate manual tasks always witness notable bottom-line growth. The market has many resources to provide the required solutions. However, each solution is designed keeping in mind the needs of a whole industry or collection of industries. The products have many features and capabilities. On the other hand, the needs of a business are usually very specific. They usually make use of a limited number of features that a provider offers. Too many features make it difficult to deliver efficiency. To quote an example, a large chain of departmental stores would require an extensive CRM software to keep track of the usage patterns its many customers. On the other hand, CRM needs a private cash-and-carry store would be limited to, say, managing contact details and email integration. In this case, buying the same software and paying the same price would be unfair and inefficient. Clearly, the best solution would be to pay for what you actually use, instead of everything that is offered by a software. Custom Solutions Some software designers offer to curate solutions to cater to the specific needs of the client. They understand the business problem at hand, restructure their product, and deliver a solution that the clients can actually use. When business interests are aligned in this manner, end users’ delight becomes possible. With the spread of cloud computing, several providers offer intuitive ready-made cloud-based pay-as-you-go-solutions. Businesses must ensure that they know how such solutions can be customized to suit their own needs. Simple solutions that actually solve problems are of much more value than fancy softwares that do not. They will meet customers’ expectations and they will have the functionalities that are actually useful. This will make work easy and infuse agility into an organization. At the end of the day, that is exactly what users want. Data Training In the artificial intelligence space, data is king. How good a solution is will depend on the kind of data it has been trained on. Its ‘ability to predict’ is directly influenced by ‘what it knows’. The nature of data which a solution is trained on determines how good the solution will be. Obviously, ready-made solutions are made using data which is available in the public domain. Such data is generic. Results from a solution which is trained on generic data are not useful for the specific needs of a business. However, there are quite a few AI based developers who train the algorithm on the client’s data. For example, if Uber would like to use AI to get insights into tweets by disgruntled users, they would want a solution that can understand feedback related phrases such as ‘app is scratchy’, ‘waited for 30 minutes’, ‘driver didn’t accept cash’ etc. A custom solutions provider will actually train the algorithms on feedback-related data specifically for the client. Support A large part of investing in a software is the process of integration. Support from the vendor is required at each and every step of the way in order to make the integration process a success. A good support service goes a long way in bringing about happy customers and, thereby, an optimistic bottom line. With custom solutions, the best part is that good customer support service is the objective of the vendor right from the beginning. The entire solution is meant to fit into the client’s needs. This makes it inevitable for the vendor to put in place a dedicated team that will ensure a seamless integration. In this case, clients will find it easy to solve their problems because help will be readily available. This feature is very particular to custom software providers and is a major boon to their clients. How ParallelDots is leading in the custom solutions space ParallelDots’ Enterprise Services team has successfully deployed AI projects across domains and industries. As leaders in the deep learning and NLP sphere, we are bringing AI right to your fingertips. Various features of ParallelDots’ Enterprise Services: EXTRAORDINARY TEAM OF DATA SCIENTISTS The ParallelDots data science team is a collection of the best lot of techies, from across the country. There is a dedicated team for data tagging as well. The purpose is to build proprietary data that can be readily used whenever a client comes aboard. DEDICATED CUSTOMER SUCCESS MANAGER The intent is to deliver solutions that actually help clients make their work easy. For this purpose, a dedicated team of customer success managers makes sure that right from integration to pilot to deployment, all processes are smoothly handled. COST-EFFECTIVE SOLUTIONS The positioning of the products is cost-effective and the pricing policy is ‘pay-as-you-go’. Users are billed only for the excess usage beyond free tier quota. The focus is on making products affordable and best-of-class. PRIMACY TO DATA PRIVACY Privacy of client data is given top priority. All of the data used to train the algorithms is proprietary and licensed. More data is constantly being generated and tagged for use by different clients. Trusted by the best Major FMCG giants including ITC are currently using , the market research product of , Inc, to automate manual tasks and get valuable insights from their data. Karna.AI ParallelDots When it comes to finding the right custom solutions, custom software development has plenty of benefits worth considering. Is it the right choice for your company? Only you will be able to decide. But, if any of the reasons we have laid out clicked in your brain as something you need, we believe your business has a great opportunity to improve and continue moving forward through the utilization of custom software development. If you want to explore or know about the custom solutions then please fill the form or simply mail at contact@paralleldots.com. We will be happy to help you! here Read more about the recently launched Google Sheets add-on for all your text analytics needs . here We hope you liked the article. Please for a free ParallelDots account to start your AI journey. You can also check demo’s of PrallelDots AI APIs . Sign Up here This article has been published after the permission of the author. You can read the original article . here

Why custom solutions work better than off-the-shelf solutions for businesses

Online data is growing exponentially every day. Businesses must stay ahead of the curve and be competitive, which can be achieved by steering away from traditional. Information should be available to executives as early as possible to keep the edge. Ensuring the data is transmitted correctly and reliably every time can be tricky. It is also critical for businesses to have access to the data across all departments. It is essential to process the data as soon as it comes in to make it quicker to turn data into insights. Apache Kafka, along with Apache Kafka Streams or Apache Flink, is turning out to be the best bet for processing, storing, and making streaming data readily available across different business units. Apache Kafka at the Core A key element in today's data streaming infrastructure is Kafka Queues. It is an effective messaging system that follows a publish-and-subscribe approach and is well known for its dependability and ability to withstand faults gracefully. This system is expertly designed to handle message volumes, per second and supports applications that rely on real-time data streaming. Apache Kafka data queues are essential for tailoring to diverse industry requirements. Kafka supports different industry scenarios, especially real-time data streaming, and ensures data accessibility for business functions. In the case of real-time streaming data, each data event can be treated as a single segment, parsed to various topics, and stored on Kafka servers. e.g., for a grocery store chain, a user e-commerce transaction can be parsed into user details, grocery store location, amount of transaction, inventory sold, and all of this in a fraction of seconds of the event, enabling the business to run analytics on a live-stream of data and re-route the exhausting inventory to the appropriate store thereby making sure stores needing more stock has it than it is sitting idle on the shelf for other stores. Apache Kafka's role extends beyond real-time streaming data. It is a vital pillar in integrating data infrastructure between business units or company mergers/acquisitions. Instead of copying data multiple times, Kafka serves as a central hub where all business components can produce data (Producers), and the business units that need it can consume it (Consumers). Kafka as a central data store helps avoid the need for repetitive engineering work by different business units to consume the same data and if needed modify on the fly using Kafka Streams or Flink thereby simplifying the data integration process. With the introduction of KRAFT in Apache Kafka, eliminating the need for a quorum manager zookeeper cluster, the cost of operating Kafka queues is even less with the same performance. Reliability, Scalability, and Security The challenge extends beyond speed; reliability, scalability, and security are equally vital. Data transmission systems should be designed for speed but other aspects of the systems can not be underestimated. These systems are required to ensure reliable delivery of millions of messages. Furthermore, scalability is imperative to handle the increasing data volumes. Reliability: A system that is high-speed but not reliable is of no use. An effective system requires monitoring and automating recovery procedures while also planning to handle the increasing amount of data effectively. It is crucial for reliability to detect and address issues before they worsen through monitoring. A monitoring strategy covering all aspects from hardware to software, to business operations, is essential. Kafka is a memory-hogging system, and enough memory should be available, as without it, the system will default to an operating system swap, thereby degrading performance. On the software level, fault tolerance is essential, which can be achieved by ensuring a minimum replication factor of three (three copies of data in a cluster). Rack diversity ensures that all three copies of data are on different rack nodes in the Kafka Cluster, thereby reducing the risk of data loss in case of a rack failure. Auto rebalancing of data based on cluster performance can be achieved using Kafka Cruise Control based on various target thresholds e.g., network bandwidth and the number of messages in or out. In the business process, enough monitoring should be added to ensure producer and consumer instances are live and able to communicate with Kafka brokers. Multiple instances of producer and consumer applications need to be implemented to ensure high availability. Scalability: The Kafka clusters must be able to handle parallel processing for both reading and writing. If the cluster struggles to keep up with read requests, it needs more resources per server - Vertical Scaling. This means using higher CPU and RAM configuration servers to handle the increased workload. If the cluster is having difficulty handling write requests, the cluster needs additional servers - Horizontal Scaling. This involves adding more servers to the cluster to share the workload. Kafka is a distributed system, scaling vertically or horizontally is a transparent operation for applications as long as applications have implemented retry on certain errors. Security: In today's digital world, data security is a top priority. Data must be encrypted at rest and in transit using the TLS protocol in Kafka. Robust user access control can be established as a two-step process using the Authentication and Authorization approach to ensure secure access. For the first step, Kafka offers multiple authentication mechanisms such as mutual TLS (mTLS) and Simple Authentication and Security Layer (SASL) to ensure that only permitted applications can authenticate to the cluster. mTLS encryption makes sure that both the client and the server authenticate each other, while SASL provides a framework for adding authentication support to connection-based protocols. For the second step, Kafka offers authorization on a topic level with appropriate privileges, such as read or write access. Overall, the Apache Kafka Ecosystem provides robust data infrastructure for processing and securely storing live data while making it readily available for all business units without reinventing the wheel of data processing. Online data is growing exponentially every day. Businesses must stay ahead of the curve and be competitive, which can be achieved by steering away from traditional. Information should be available to executives as early as possible to keep the edge. Ensuring the data is transmitted correctly and reliably every time can be tricky. It is also critical for businesses to have access to the data across all departments. It is essential to process the data as soon as it comes in to make it quicker to turn data into insights. Apache Kafka, along with Apache Kafka Streams or Apache Flink, is turning out to be the best bet for processing, storing, and making streaming data readily available across different business units. Apache Kafka at the Core Apache Kafka at the Core A key element in today's data streaming infrastructure is Kafka Queues. It is an effective messaging system that follows a publish-and-subscribe approach and is well known for its dependability and ability to withstand faults gracefully. This system is expertly designed to handle message volumes, per second and supports applications that rely on real-time data streaming. Apache Kafka data queues are essential for tailoring to diverse industry requirements. Kafka supports different industry scenarios, especially real-time data streaming, and ensures data accessibility for business functions. In the case of real-time streaming data, each data event can be treated as a single segment, parsed to various topics, and stored on Kafka servers. e.g., for a grocery store chain, a user e-commerce transaction can be parsed into user details, grocery store location, amount of transaction, inventory sold, and all of this in a fraction of seconds of the event, enabling the business to run analytics on a live-stream of data and re-route the exhausting inventory to the appropriate store thereby making sure stores needing more stock has it than it is sitting idle on the shelf for other stores. Apache Kafka's role extends beyond real-time streaming data. It is a vital pillar in integrating data infrastructure between business units or company mergers/acquisitions. Instead of copying data multiple times, Kafka serves as a central hub where all business components can produce data (Producers), and the business units that need it can consume it (Consumers). Kafka as a central data store helps avoid the need for repetitive engineering work by different business units to consume the same data and if needed modify on the fly using Kafka Streams or Flink thereby simplifying the data integration process. With the introduction of KRAFT in Apache Kafka, eliminating the need for a quorum manager zookeeper cluster, the cost of operating Kafka queues is even less with the same performance. Reliability, Scalability, and Security Reliability, Scalability, and Security The challenge extends beyond speed; reliability, scalability, and security are equally vital. Data transmission systems should be designed for speed but other aspects of the systems can not be underestimated. These systems are required to ensure reliable delivery of millions of messages. Furthermore, scalability is imperative to handle the increasing data volumes. Reliability: Reliability: A system that is high-speed but not reliable is of no use. An effective system requires monitoring and automating recovery procedures while also planning to handle the increasing amount of data effectively. It is crucial for reliability to detect and address issues before they worsen through monitoring. A monitoring strategy covering all aspects from hardware to software, to business operations, is essential. Kafka is a memory-hogging system, and enough memory should be available, as without it, the system will default to an operating system swap, thereby degrading performance. On the software level, fault tolerance is essential, which can be achieved by ensuring a minimum replication factor of three (three copies of data in a cluster). Rack diversity ensures that all three copies of data are on different rack nodes in the Kafka Cluster, thereby reducing the risk of data loss in case of a rack failure. Auto rebalancing of data based on cluster performance can be achieved using Kafka Cruise Control based on various target thresholds e.g., network bandwidth and the number of messages in or out. In the business process, enough monitoring should be added to ensure producer and consumer instances are live and able to communicate with Kafka brokers. Multiple instances of producer and consumer applications need to be implemented to ensure high availability. Scalability: Scalability: The Kafka clusters must be able to handle parallel processing for both reading and writing. If the cluster struggles to keep up with read requests, it needs more resources per server - Vertical Scaling. This means using higher CPU and RAM configuration servers to handle the increased workload. If the cluster is having difficulty handling write requests, the cluster needs additional servers - Horizontal Scaling. This involves adding more servers to the cluster to share the workload. Kafka is a distributed system, scaling vertically or horizontally is a transparent operation for applications as long as applications have implemented retry on certain errors. Security: Security: In today's digital world, data security is a top priority. Data must be encrypted at rest and in transit using the TLS protocol in Kafka. Robust user access control can be established as a two-step process using the Authentication and Authorization approach to ensure secure access. For the first step, Kafka offers multiple authentication mechanisms such as mutual TLS (mTLS) and Simple Authentication and Security Layer (SASL) to ensure that only permitted applications can authenticate to the cluster. mTLS encryption makes sure that both the client and the server authenticate each other, while SASL provides a framework for adding authentication support to connection-based protocols. For the second step, Kafka offers authorization on a topic level with appropriate privileges, such as read or write access. Overall, the Apache Kafka Ecosystem provides robust data infrastructure for processing and securely storing live data while making it readily available for all business units without reinventing the wheel of data processing.

Mastering the Complexity of High-Volume Data Transmission in the Digital Age

Authors:
(1) Haleh Hayati, Department of Mechanical Engineering, Dynamics and Control Group, Eindhoven University of Technology, The Netherlands;
(2) Nathan van de Wouw, Department of Mechanical Engineering, Dynamics and Control Group, Eindhoven University of Technology, The Netherlands;
(3) Carlos Murguia, Department of Mechanical Engineering, Dynamics and Control Group, Eindhoven University of Technology, The Netherlands, and with the School of Electrical Engineering and Robotics, Queensland University of Technology, Brisbane, Australia. Table of Links Abstract and Introduction Problem Formulation Affine Solution to Problem 1 Privacy Guarantee General Guidelines for Implementation Case Studies Conclusion and References IV. PRIVACY GUARANTEES A privacy-level analysis of random affine maps of the form (7) has been discussed in the literature, see [33] (cryptanalysis) and [34] (DP). However, the guarantees they show are for square and invertible encoding matrices. In our scheme, we cannot directly apply those results as our encoding matrices (Π1 and Π3) are rectangular (tall matrices). We need rectangular matrices to be able to exploit their kernels to remove the injected distortion at the user side. A complication with this architecture is that the covariance matrix of N1sk (which is part of the user data encoding mechanism) is degenerate regardless of the probability distribution of sk. The latter prevents us from using off-the-shelf tools to design the affine maps to guarantee differential privacy. In what follows, we provide a tailored solution to guarantee DP for the class of mechanisms that we consider. In particular, we prove that the proposed scheme, with full-column rank encoding matrices, can provide any desired level of differential privacy without reducing the accuracy and performance of the original algorithm. A. Differential Privacy Definition 1 [Adjacency] Let X denote the space of all possible datasets. We say that D ∈ X and D′ ∈ X are adjacent if they differ on a single element. Definition 2 [(ϵ, δ)-Differential Privacy [35]] The random mechanism M : X → R with domain X and range R is said to provide (ϵ, δ)-differential privacy, if for any two adjacent datasets D, D′ ∈ X and for all measurable sets S ⊆ R: If δ = 0, M is said to satisfy ϵ-differential privacy. From Definition 2, we have that a mechanism provides DP if its probability distribution satisfies (25) for some ϵ and δ. Then, if we seek to design the mechanism to guarantee DP, we need to shape its probability distribution. This is usually done by injecting noise into the data we seek to encode. The noise statistics must be designed in terms of the sensitivity of the data to be encoded. Sensitivity refers to the maximum change of the data due to the difference in a single element of the dataset. Definition 3 (Sensitivity) : Given two adjacent datasets D, D′ ∈ X , and a query function q : X → R (a deterministic function of datasets). The sensitivity of q(·) is formulated as A differential privacy mechanism M based on the sensitivity (26) needs to be designed so that the differential privacy condition (25) holds. B. Immersion-based Coding Differential Privacy Guarantee Solution to Problem 3 where (a) follows from the following triangle inequality Due to the sensitivity relation (28), we have Hence, inequality (30) implies C. Perfect Security A perfectly secure cryptosystem is a secure system against adversaries with unlimited computing resources and time, see [36]. In [37], Shannon proves that a necessary condition encryption method to be perfectly secure is that the uncertainty of the secret key is larger than or equal to the uncertainty of the plaintext, see [38]. He proposes a one-time pad encryption scheme in which the key is randomly selected and never used again. The one-time pad gives unbounded entropy of the key space, i.e., infinite key space, which provides perfect security. Perfect secrecy is lost when the key is not random or if it is reused. This perfect security idea can be used to assess the performance of our proposed coding mechanism. Given the fact that in the proposed mechanism, (7), the encoding keys, (Π1, sk) and (Π3, Π4N1sk), are random, change in every iteration, and provide an infinite-dimensional key space (the support of sk), can be considered perfectly secure. Perfect secrecy of the proposed coding mechanisms can also be concluded from the differential privacy point of view. In [39], the authors define perfect secrecy in terms of DP. They state that a mechanism M : X → R, with domain X and range R is perfectly secret if for any two adjacent D, D′ ∈ X and for all measurable sets S ⊆ R. Then, the output of the mechanism reveals no information about private data. Comparing the definition of perfect secrecy in (37) and the definition of differential privacy (Definition 2), we can conclude that a differentially private mechanism is perfectly secret if (ϵ, δ) equal zero. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Haleh Hayati, Department of Mechanical Engineering, Dynamics and Control Group, Eindhoven University of Technology, The Netherlands; (2) Nathan van de Wouw, Department of Mechanical Engineering, Dynamics and Control Group, Eindhoven University of Technology, The Netherlands; (3) Carlos Murguia, Department of Mechanical Engineering, Dynamics and Control Group, Eindhoven University of Technology, The Netherlands, and with the School of Electrical Engineering and Robotics, Queensland University of Technology, Brisbane, Australia. Authors: Authors: (1) Haleh Hayati, Department of Mechanical Engineering, Dynamics and Control Group, Eindhoven University of Technology, The Netherlands; (2) Nathan van de Wouw, Department of Mechanical Engineering, Dynamics and Control Group, Eindhoven University of Technology, The Netherlands; (3) Carlos Murguia, Department of Mechanical Engineering, Dynamics and Control Group, Eindhoven University of Technology, The Netherlands, and with the School of Electrical Engineering and Robotics, Queensland University of Technology, Brisbane, Australia. Table of Links Abstract and Introduction Abstract and Introduction Problem Formulation Problem Formulation Affine Solution to Problem 1 Affine Solution to Problem 1 Privacy Guarantee Privacy Guarantee General Guidelines for Implementation General Guidelines for Implementation Case Studies Case Studies Conclusion and References Conclusion and References IV. PRIVACY GUARANTEES A privacy-level analysis of random affine maps of the form (7) has been discussed in the literature, see [33] (cryptanalysis) and [34] (DP). However, the guarantees they show are for square and invertible encoding matrices. In our scheme, we cannot directly apply those results as our encoding matrices (Π1 and Π3) are rectangular (tall matrices). We need rectangular matrices to be able to exploit their kernels to remove the injected distortion at the user side. A complication with this architecture is that the covariance matrix of N1sk (which is part of the user data encoding mechanism) is degenerate regardless of the probability distribution of sk. The latter prevents us from using off-the-shelf tools to design the affine maps to guarantee differential privacy. In what follows, we provide a tailored solution to guarantee DP for the class of mechanisms that we consider. In particular, we prove that the proposed scheme, with full-column rank encoding matrices, can provide any desired level of differential privacy without reducing the accuracy and performance of the original algorithm. A. Differential Privacy A. Differential Privacy Definition 1 [Adjacency] Let X denote the space of all possible datasets. We say that D ∈ X and D′ ∈ X are adjacent if they differ on a single element. Definition 1 [Adjacency] Let X denote the space of all possible datasets. We say that D ∈ X and D′ ∈ X are adjacent if they differ on a single element. Definition 2 [(ϵ, δ)-Differential Privacy [35]] The random mechanism M : X → R with domain X and range R is said to provide (ϵ, δ)-differential privacy, if for any two adjacent datasets D, D′ ∈ X and for all measurable sets S ⊆ R: Definition 2 [(ϵ, δ)-Differential Privacy [35]] The random mechanism M : X → R with domain X and range R is said to provide (ϵ, δ)-differential privacy, if for any two adjacent datasets D, D′ ∈ X and for all measurable sets S ⊆ R: If δ = 0, M is said to satisfy ϵ-differential privacy. From Definition 2, we have that a mechanism provides DP if its probability distribution satisfies (25) for some ϵ and δ. Then, if we seek to design the mechanism to guarantee DP, we need to shape its probability distribution. This is usually done by injecting noise into the data we seek to encode. The noise statistics must be designed in terms of the sensitivity of the data to be encoded. Sensitivity refers to the maximum change of the data due to the difference in a single element of the dataset. Definition 3 (Sensitivity) : Given two adjacent datasets D, D′ ∈ X , and a query function q : X → R (a deterministic function of datasets). The sensitivity of q(·) is formulated as Definition 3 (Sensitivity) : Given two adjacent datasets D, D′ ∈ X , and a query function q : X → R (a deterministic function of datasets). The sensitivity of q(·) is formulated as A differential privacy mechanism M based on the sensitivity (26) needs to be designed so that the differential privacy condition (25) holds. B. Immersion-based Coding Differential Privacy Guarantee B. Immersion-based Coding Differential Privacy Guarantee Solution to Problem 3 Solution to Problem 3 where (a) follows from the following triangle inequality Due to the sensitivity relation (28), we have Hence, inequality (30) implies C. Perfect Security C. Perfect Security A perfectly secure cryptosystem is a secure system against adversaries with unlimited computing resources and time, see [36]. In [37], Shannon proves that a necessary condition encryption method to be perfectly secure is that the uncertainty of the secret key is larger than or equal to the uncertainty of the plaintext, see [38]. He proposes a one-time pad encryption scheme in which the key is randomly selected and never used again. The one-time pad gives unbounded entropy of the key space, i.e., infinite key space, which provides perfect security. Perfect secrecy is lost when the key is not random or if it is reused. This perfect security idea can be used to assess the performance of our proposed coding mechanism. Given the fact that in the proposed mechanism, (7), the encoding keys, (Π1, sk) and (Π3, Π4N1sk), are random, change in every iteration, and provide an infinite-dimensional key space (the support of sk), can be considered perfectly secure. Perfect secrecy of the proposed coding mechanisms can also be concluded from the differential privacy point of view. In [39], the authors define perfect secrecy in terms of DP. They state that a mechanism M : X → R, with domain X and range R is perfectly secret if for any two adjacent D, D′ ∈ X and for all measurable sets S ⊆ R. Then, the output of the mechanism reveals no information about private data. Comparing the definition of perfect secrecy in (37) and the definition of differential privacy (Definition 2), we can conclude that a differentially private mechanism is perfectly secret if (ϵ, δ) equal zero. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Privacy in Cloud Computing through Immersion-based Coding: Privacy Guarantee

(1) Feng Liang, The University of Texas at Austin and Work partially done during an internship at Meta GenAI (Email: jeffliang@utexas.edu);
(2) Bichen Wu, Meta GenAI and Corresponding author;
(3) Jialiang Wang, Meta GenAI;
(4) Licheng Yu, Meta GenAI;
(5) Kunpeng Li, Meta GenAI;
(6) Yinan Zhao, Meta GenAI;
(7) Ishan Misra, Meta GenAI;
(8) Jia-Bin Huang, Meta GenAI;
(9) Peizhao Zhang, Meta GenAI (Email: stzpz@meta.com);
(10) Peter Vajda, Meta GenAI (Email: vajdap@meta.com);
(11) Diana Marculescu, The University of Texas at Austin (Email: dianam@utexas.edu). Table of Links Abstract and Introduction
2. Related Work
3. Preliminary
4. FlowVid
4.1. Inflating image U-Net to accommodate video
4.2. Training with joint spatial-temporal conditions
4.3. Generation: edit the first frame then propagate


Experiments


5.1. Settings
5.2. Qualitative results
5.3. Quantitative results
5.4. Ablation study and 5.5. Limitations
Conclusion, Acknowledgments and References
A. Webpage Demo and B. Quantitative comparisons 5. Experiments 5.1. Settings Implementation Details We train our model with 100k videos from Shutterstock [1]. For each training video, we sequentially sample 16 frames with interval {2,4,8}, which represent videos lasting {1,2,4} seconds (taking videos with FPS of 30). The resolution of all images, including input frames, spatial condition images, and flow warped frames, is set to 512×512 via center crop. We train the model with a batch size of 1 per GPU and a total batch size of 8 with 8 GPUs. We employ AdamW optimizer [28] with a learning rate of 1e-5 for 100k iterations. As detailed in our method, we train the major U-Net and ControlNet U-Net joint branches with v-parameterization [41]. The training takes four days on one 8-A100-80G node. During generation, we first generate keyframes with our trained model and then use an off-the-shelf frame interpolation model, such as RIFE [21], to generate non-key frames. By default, we produce 16 key frames at an interval of 4, corresponding to a 2-second clip at 8 FPS. Then, we use RIFE to interpolate the results to 32 FPS. We employ classifier-free guidance [15] with a scale of 7.5 and use 20 inference sampling steps. Additionally, the Zero SNR noise scheduler [27] is utilized. We also fuse the self-attention features obtained during the DDIM inversion of corresponding key frames from the input video, following FateZero [35]. We evaluate our FlowVid with two different spatial conditions: canny edge maps [5] and depth maps [38]. A comparison of these controls can be found in Section 5.4. Evaluation We select the 25 object-centric videos from the public DAVIS dataset [34], covering humans, animals, etc We manually design 115 prompts for these videos, spanning from stylization to object swap. Besides, we also collect 50 Shutterstock videos [1] with 200 designed prompts. We conduct both qualitative (see Section 5.2) and quantitative comparisons (see Section 5.3) with state-of-the-art methods including Rerender [49], CoDeF [32] and TokenFlow [13]. We use their official codes with the default settings. This paper is available on arxiv under CC 4.0 license. (1) Feng Liang, The University of Texas at Austin and Work partially done during an internship at Meta GenAI (Email: jeffliang@utexas.edu); (2) Bichen Wu, Meta GenAI and Corresponding author; (3) Jialiang Wang, Meta GenAI; (4) Licheng Yu, Meta GenAI; (5) Kunpeng Li, Meta GenAI; (6) Yinan Zhao, Meta GenAI; (7) Ishan Misra, Meta GenAI; (8) Jia-Bin Huang, Meta GenAI; (9) Peizhao Zhang, Meta GenAI (Email: stzpz@meta.com); (10) Peter Vajda, Meta GenAI (Email: vajdap@meta.com); (11) Diana Marculescu, The University of Texas at Austin (Email: dianam@utexas.edu). (1) Feng Liang, The University of Texas at Austin and Work partially done during an internship at Meta GenAI (Email: jeffliang@utexas.edu); (2) Bichen Wu, Meta GenAI and Corresponding author; (3) Jialiang Wang, Meta GenAI; (4) Licheng Yu, Meta GenAI; (5) Kunpeng Li, Meta GenAI; (6) Yinan Zhao, Meta GenAI; (7) Ishan Misra, Meta GenAI; (8) Jia-Bin Huang, Meta GenAI; (9) Peizhao Zhang, Meta GenAI (Email: stzpz@meta.com); (10) Peter Vajda, Meta GenAI (Email: vajdap@meta.com); (11) Diana Marculescu, The University of Texas at Austin (Email: dianam@utexas.edu). Table of Links Abstract and Introduction 2. Related Work 3. Preliminary 4. FlowVid 4.1. Inflating image U-Net to accommodate video 4.2. Training with joint spatial-temporal conditions 4.3. Generation: edit the first frame then propagate Experiments 5.1. Settings 5.2. Qualitative results 5.3. Quantitative results 5.4. Ablation study and 5.5. Limitations Conclusion, Acknowledgments and References A. Webpage Demo and B. Quantitative comparisons Abstract and Introduction Abstract and Introduction 2. Related Work 2. Related Work 3. Preliminary 3. Preliminary 4. FlowVid 4. FlowVid 4 .1. Inflating image U-Net to accommodate video .1. Inflating image U-Net to accommodate video 4.2. Training with joint spatial-temporal conditions 4.2. Training with joint spatial-temporal conditions 4.3. Generation: edit the first frame then propagate 4.3. Generation: edit the first frame then propagate Experiments Experiments Experiments 5.1. Settings 5.1. Settings 5.2. Qualitative results 5.2. Qualitative results 5.3. Quantitative results 5.3. Quantitative results 5.4. Ablation study and 5.5. Limitations 5.4. Ablation study and 5.5. Limitations Conclusion, Acknowledgments and References Conclusion, Acknowledgments and References A. Webpage Demo and B. Quantitative comparisons A. Webpage Demo and B. Quantitative comparisons 5. Experiments 5.1. Settings Implementation Details We train our model with 100k videos from Shutterstock [1]. For each training video, we sequentially sample 16 frames with interval {2,4,8}, which represent videos lasting {1,2,4} seconds (taking videos with FPS of 30). The resolution of all images, including input frames, spatial condition images, and flow warped frames, is set to 512×512 via center crop. We train the model with a batch size of 1 per GPU and a total batch size of 8 with 8 GPUs. We employ AdamW optimizer [28] with a learning rate of 1e-5 for 100k iterations. As detailed in our method, we train the major U-Net and ControlNet U-Net joint branches with v-parameterization [41]. The training takes four days on one 8-A100-80G node. Implementation Details During generation, we first generate keyframes with our trained model and then use an off-the-shelf frame interpolation model, such as RIFE [21], to generate non-key frames. By default, we produce 16 key frames at an interval of 4, corresponding to a 2-second clip at 8 FPS. Then, we use RIFE to interpolate the results to 32 FPS. We employ classifier-free guidance [15] with a scale of 7.5 and use 20 inference sampling steps. Additionally, the Zero SNR noise scheduler [27] is utilized. We also fuse the self-attention features obtained during the DDIM inversion of corresponding key frames from the input video, following FateZero [35]. We evaluate our FlowVid with two different spatial conditions: canny edge maps [5] and depth maps [38]. A comparison of these controls can be found in Section 5.4. Evaluation We select the 25 object-centric videos from the public DAVIS dataset [34], covering humans, animals, etc We manually design 115 prompts for these videos, spanning from stylization to object swap. Besides, we also collect 50 Shutterstock videos [1] with 200 designed prompts. We conduct both qualitative (see Section 5.2) and quantitative comparisons (see Section 5.3) with state-of-the-art methods including Rerender [49], CoDeF [32] and TokenFlow [13]. We use their official codes with the default settings. Evaluation This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis: Settings

AI Fundamentals: A Practical Guide for All Professionals

Authors:
(1) Yinwei Dai, Princeton University (Equal contributions);
(2) Rui Pan, Princeton University (Equal contributions);
(3) Anand Iyer, Georgia Institute of Technology;
(4) Ravi Netravali, Georgia Institute of Technology. Table of Links Abstract and 1 Introduction 2 Background and Motivation and 2.1 Model Serving Platforms 2.2 Early-Exit Models 2.3 Challenges 3 Design 3.1 Preparing Models with Early Exits 3.2 Accuracy-Aware Threshold Tuning 3.3 Latency-Focused Ramp Adjustments 4 Implementation 5 Evaluation and 5.1 Methodology 5.2 Overall Results 5.3 Comparison with Existing EE Strategies 5.4 Microbenchmarks 6 Additional Related Work 7 Conclusion, References, Appendix 5.3 Comparison with Existing EE Strategies We compare Apparate with two off-the-shelf EE models: BranchyNet [53] and DeeBERT [57]. BranchyNet extends ResNet models with ramps of the same style as Apparate, while DeeBERT extends BERT-base with deeper ramps (using the entire BERT pooler, as described in §3.1). For each, we follow their prescribed architectures, with ramps after every layer that are always active. We perform one-time tuning of thresholds as recommended by both works, and consider two variants: the default recommendation where all ramps must use the same threshold, and a more flexible version that removes this restriction (+). For both, threshold tuning is done optimally (via grid search), and is based on uniformly sampled data across the workload. For fair comparison, Apparate’s ramp budget is configured to support ramps at all layers (though it never does so). Table 2 presents our results. The main takeaway is that existing EE approaches, even when favorably tuned, yield unacceptable drops in average accuracy up to 23.9% and 17.8% for CV and NLP. In contrast, Apparate consistently meets the imposed accuracy constraint (1% in this experiment) for both workloads. Further, even with such accuracy violations, tail latencies are 0.9-9.4% lower with Apparate than with these systems. The reason is again lack of adaptation: all ramps are always active despite their current efficacy which vary dramatically over time (§2.3), yielding undue overheads for large numbers of non-exiting inputs. In contrast, throughout these experiments, despite having a full ramp budget, Apparate maintained only 9.1-27.2% of all possible ramps. For fair median latency comparison, we consider an optimally-tuned (opt) version of existing EE models that perform one-time tuning on the actual test dataset, picking the best (latency-wise) thresholds that ensure <1% accuracy drop. As shown, due to its regular and less-constrained adaptation, Apparate outperforms even this oracle version of existing EEs with up to 14.1% higher median latency savings. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Yinwei Dai, Princeton University (Equal contributions); (2) Rui Pan, Princeton University (Equal contributions); (3) Anand Iyer, Georgia Institute of Technology; (4) Ravi Netravali, Georgia Institute of Technology. Authors: Authors: (1) Yinwei Dai, Princeton University (Equal contributions); (2) Rui Pan, Princeton University (Equal contributions); (3) Anand Iyer, Georgia Institute of Technology; (4) Ravi Netravali, Georgia Institute of Technology. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Background and Motivation and 2.1 Model Serving Platforms 2 Background and Motivation and 2.1 Model Serving Platforms 2.2 Early-Exit Models 2.2 Early-Exit Models 2.3 Challenges 2.3 Challenges 3 Design 3 Design 3.1 Preparing Models with Early Exits 3.1 Preparing Models with Early Exits 3.2 Accuracy-Aware Threshold Tuning 3.2 Accuracy-Aware Threshold Tuning 3.3 Latency-Focused Ramp Adjustments 3.3 Latency-Focused Ramp Adjustments 4 Implementation 4 Implementation 5 Evaluation and 5.1 Methodology 5 Evaluation and 5.1 Methodology 5.2 Overall Results 5.2 Overall Results 5.3 Comparison with Existing EE Strategies 5.3 Comparison with Existing EE Strategies 5.4 Microbenchmarks 5.4 Microbenchmarks 6 Additional Related Work 6 Additional Related Work 7 Conclusion, References, Appendix 7 Conclusion, References, Appendix 5.3 Comparison with Existing EE Strategies We compare Apparate with two off-the-shelf EE models: BranchyNet [53] and DeeBERT [57]. BranchyNet extends ResNet models with ramps of the same style as Apparate, while DeeBERT extends BERT-base with deeper ramps (using the entire BERT pooler, as described in §3.1). For each, we follow their prescribed architectures, with ramps after every layer that are always active. We perform one-time tuning of thresholds as recommended by both works, and consider two variants: the default recommendation where all ramps must use the same threshold, and a more flexible version that removes this restriction (+). For both, threshold tuning is done optimally (via grid search), and is based on uniformly sampled data across the workload. For fair comparison, Apparate’s ramp budget is configured to support ramps at all layers (though it never does so). Table 2 presents our results. The main takeaway is that existing EE approaches, even when favorably tuned, yield unacceptable drops in average accuracy up to 23.9% and 17.8% for CV and NLP. In contrast, Apparate consistently meets the imposed accuracy constraint (1% in this experiment) for both workloads. Further, even with such accuracy violations, tail latencies are 0.9-9.4% lower with Apparate than with these systems. The reason is again lack of adaptation: all ramps are always active despite their current efficacy which vary dramatically over time (§2.3), yielding undue overheads for large numbers of non-exiting inputs. In contrast, throughout these experiments, despite having a full ramp budget, Apparate maintained only 9.1-27.2% of all possible ramps. For fair median latency comparison, we consider an optimally-tuned (opt) version of existing EE models that perform one-time tuning on the actual test dataset, picking the best (latency-wise) thresholds that ensure <1% accuracy drop. As shown, due to its regular and less-constrained adaptation, Apparate outperforms even this oracle version of existing EEs with up to 14.1% higher median latency savings. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Apparate: Early-Exit Models for ML Latency and Throughput Optimization - Comparisons

As a developer, frameworks are usually the first thing you grab when you want to speed things up and keep things reliable. People often talk about frameworks like they’re the perfect solution that can fix all your problems, making development faster, easier, and more efficient. However, if you've had some experience under your belt, you know that frameworks aren’t a one-size-fits-all solution. Picking the right one can streamline your work, but the wrong choice can lead to headaches down the road, slowing you down just when you need to move quickly. In this blog post, we’re going to dive into the real challenges and strategies that come with choosing and using frameworks. We’ll look at the potential pitfalls, how to avoid them, and ways to keep your codebase flexible — even when a framework is in play. The Relationship You Didn’t Know You Were Getting Into Committing to a framework is a bit like getting into a long-term relationship. And it’s not something to take lightly. Unlike a simple library or a small utility, frameworks come with opinions — lots of them. They impose structure and methodology on your application, whether you like it or not. It's crucial to remember that framework creators have their own priorities. They’re solving THEIR problems, not yours. They don’t owe you anything (unless, of course, you’ve got a buddy on the internal framework team, in which case, lucky you). If things go south, especially deep into your project, you could be in for a world of hurt. Now you’re stuck fixing it, or worse, ripping it out entirely. Not fun, right? So before you tie yourself to a framework, make sure it actually fits what you need. Otherwise, you’re rolling the dice. FAANG Problems Author has no idea if FAANG became MAANG, MANGA, or if we're all just in an anime now. Here’s where experience really counts. When companies grow rapidly, they often face challenges that no off-the-shelf solution can handle. The scale of these problems forces them to create their own tools — custom databases, ETL engines, BI tools — you name it. Big Tech giants like Google, LinkedIn, Spotify, and Netflix have led the way, building and open-sourcing tools that the rest of us now get to play with. But here’s the thing: these tools weren’t built to be universally applicable. They were created to solve specific problems that most companies will never encounter. Engineers who’ve worked at these big companies are used to dealing with these kinds of challenges — they’ve built solutions that operate at a scale most of us can only imagine. So when they move to smaller companies, the framework and tooling decisions they make are based on a deep understanding of both the power and the pitfalls of these technologies. Pushback Against Frameworks Lately, there’s been a bit of a rebellion brewing—people are getting tired of frameworks. Especially in the JavaScript world, developers are fed up with the constant churn. You’ve probably seen it: every time a major update drops, you have to rewrite significant chunks of your codebase just to stay relevant. And don’t get me started on the endless cycle of breaking changes. This frustration has given rise to a revival of simpler, more stable stacks. Stuff like vanilla HTML, CSS, jQuery, PHP, and SQLite are making a comeback among developers who prioritize getting things done over staying on the bleeding edge of tech. Yeah, it might feel a bit “old-school,” but it’s far from obsolete. With a simpler stack, you can iterate fast and ship even faster. Sure, newer frameworks like React, Node.js, and Flask have their place, but sometimes you don’t need all the fancy stuff. Sometimes, sticking to what works can save you a whole lot of headaches. The Framework Industrial Complex? Are frameworks becoming... overhyped? It’s hard not to notice that some frameworks feel more like tools designed to lure in VC funding than to solve real developer problems. It’s like there’s a whole ecosystem pushing developers into these frameworks, only for them to realize later that, once they scale, they’re locked into expensive platforms. Sure, frameworks like Databricks are open-source and free to start with, but as you grow, you're nudged towards their enterprise solutions. And suddenly, your hosting and operational costs are through the roof, while a simple VPS might have sufficed. It feels a bit like a trap, doesn’t it? Delaying the Framework Decision Here’s a piece of advice that I swear by: don’t rush into choosing a framework. Don’t commit until your architecture is fully fleshed out. The framework should be the last thing you worry about, not the first. First, make sure your architecture is solid. Know your core components and how they’ll interact. Once you’ve got that, you can evaluate frameworks with a clear understanding of where they might fit — or if they even fit at all. This approach ensures that your design is solid and suited to your specific needs. When it comes time to consider a framework, you’ll be able to see clearly where it can enhance your architecture without limiting it. Before you jump into using any framework, ask yourself: do you really, truly need it? Sure, frameworks can add layers of automation and convenience, but they also come with their own set of limitations. If your application has unique requirements, frameworks might not play nice with them. Think long and hard about the long-term benefits against the limitations. Making Frameworks Expendable If you decide a framework is worth the risk, make sure it’s easy to replace. Yes, you heard that right. Build in some flexibility so that if you need to ditch it later, it’s not a monumental task. Here’s how: Abstract Your Dependencies Keep the framework’s grubby little hands out of your core code. Use interfaces to abstract the framework’s functionality so that your business logic doesn’t depend on the framework directly. Let’s say you’re using TensorFlow for machine learning. Instead of embedding TensorFlow code throughout your entire application, define interfaces to keep things neat and abstract: from abc import ABC, abstractmethod
import tensorflow as tf

class ModelTrainer(ABC):
    @abstractmethod
    def train(self, data):
        pass

class TensorFlowTrainer(ModelTrainer):
    def train(self, data):
        # TensorFlow-specific training logic
        model = tf.keras.models.Sequential([...])
        model.compile(optimizer='adam', loss='sparse_categorical_crossentropy')
        model.fit(data, epochs=5)
        return model By doing this, your core logic isn’t tightly coupled with TensorFlow. If you need to switch to another machine learning framework, it’s just a matter of swapping out the implementation. Dependency Injection (DI) is Your Friend Next, let’s talk about Dependency Injection (DI). This technique lets you inject specific implementations of your interfaces into your classes, keeping your codebase decoupled and modular. class TrainingPipeline:
    def __init__(self, trainer: ModelTrainer):
        self.trainer = trainer

    def execute(self, data):
        return self.trainer.train(data)

# Inject the TensorFlowTrainer implementation
pipeline = TrainingPipeline(TensorFlowTrainer()) Now your code is flexible, easy to test, and ready for whatever the future throws at it. Embrace Inversion of Control (IoC) For the ultimate in flexibility, take things up a notch with Inversion of Control (IoC). This pattern allows you to specify implementations in a configuration file or a centralized location in your code. It’s the cherry on top of your framework-agnostic architecture. Here’s an example of how that might work with a configuration-based approach: # config.py
class Config:
    TRAINER = 'my_project.trainers.TensorFlowTrainer'

# main.py
import importlib

class TrainingPipeline:
    def __init__(self, trainer_class: str):
        module_name, class_name = trainer_class.rsplit('.', 1)
        module = importlib.import_module(module_name)
        trainer_cls = getattr(module, class_name)
        self.trainer = trainer_cls()

    def execute(self, data):
        return self.trainer.train(data)

# Inject the trainer specified in the configuration
from config import Config
pipeline = TrainingPipeline(Config.TRAINER) Now, if you ever need to replace TensorFlow with another machine learning framework, you simply update the configuration and carry on. No hassle, no drama. Conclusion Remember, frameworks should serve YOUR architecture, not dictate it. With careful planning and strategic abstraction, you can reap the benefits of frameworks without getting trapped in long-term dependencies. The trick is staying in control. So next time you’re about to dive into a framework, take a step back and remind yourself: you call the shots here. Thank you for reading! Any questions? Leave your comment below to start fantastic discussions! Check out my blog or come to say hi 👋 on Twitter or subscribe to my telegram channel. Plan your best! As a developer, frameworks are usually the first thing you grab when you want to speed things up and keep things reliable. People often talk about frameworks like they’re the perfect solution that can fix all your problems, making development faster, easier, and more efficient. However, if you've had some experience under your belt, you know that frameworks aren’t a one-size-fits-all solution. Picking the right one can streamline your work, but the wrong choice can lead to headaches down the road, slowing you down just when you need to move quickly. In this blog post, we’re going to dive into the real challenges and strategies that come with choosing and using frameworks. We’ll look at the potential pitfalls, how to avoid them, and ways to keep your codebase flexible — even when a framework is in play. The Relationship You Didn’t Know You Were Getting Into Committing to a framework is a bit like getting into a long-term relationship. And it’s not something to take lightly. Unlike a simple library or a small utility, frameworks come with opinions — lots of them. They impose structure and methodology on your application, whether you like it or not. It's crucial to remember that framework creators have their own priorities. They’re solving THEIR problems, not yours. They don’t owe you anything (unless, of course, you’ve got a buddy on the internal framework team, in which case, lucky you). If things go south, especially deep into your project, you could be in for a world of hurt. Now you’re stuck fixing it, or worse, ripping it out entirely. Not fun, right? So before you tie yourself to a framework, make sure it actually fits what you need. Otherwise, you’re rolling the dice. FAANG Problems Author has no idea if FAANG became MAANG, MANGA, or if we're all just in an anime now. Author has no idea if FAANG became MAANG, MANGA, or if we're all just in an anime now. Here’s where experience really counts. When companies grow rapidly, they often face challenges that no off-the-shelf solution can handle. The scale of these problems forces them to create their own tools — custom databases, ETL engines, BI tools — you name it. Big Tech giants like Google, LinkedIn, Spotify, and Netflix have led the way, building and open-sourcing tools that the rest of us now get to play with. But here’s the thing: these tools weren’t built to be universally applicable. They were created to solve specific problems that most companies will never encounter. Engineers who’ve worked at these big companies are used to dealing with these kinds of challenges — they’ve built solutions that operate at a scale most of us can only imagine. So when they move to smaller companies, the framework and tooling decisions they make are based on a deep understanding of both the power and the pitfalls of these technologies. Pushback Against Frameworks Lately, there’s been a bit of a rebellion brewing—people are getting tired of frameworks. Especially in the JavaScript world, developers are fed up with the constant churn. You’ve probably seen it: every time a major update drops, you have to rewrite significant chunks of your codebase just to stay relevant. And don’t get me started on the endless cycle of breaking changes. This frustration has given rise to a revival of simpler, more stable stacks. Stuff like vanilla HTML, CSS, jQuery, PHP, and SQLite are making a comeback among developers who prioritize getting things done over staying on the bleeding edge of tech. Yeah, it might feel a bit “old-school,” but it’s far from obsolete. With a simpler stack, you can iterate fast and ship even faster. Sure, newer frameworks like React, Node.js, and Flask have their place, but sometimes you don’t need all the fancy stuff. Sometimes, sticking to what works can save you a whole lot of headaches. The Framework Industrial Complex? Are frameworks becoming... overhyped? It’s hard not to notice that some frameworks feel more like tools designed to lure in VC funding than to solve real developer problems. It’s like there’s a whole ecosystem pushing developers into these frameworks, only for them to realize later that, once they scale, they’re locked into expensive platforms. Sure, frameworks like Databricks are open-source and free to start with, but as you grow, you're nudged towards their enterprise solutions. And suddenly, your hosting and operational costs are through the roof, while a simple VPS might have sufficed. It feels a bit like a trap, doesn’t it? Delaying the Framework Decision Here’s a piece of advice that I swear by: don’t rush into choosing a framework . Don’t commit until your architecture is fully fleshed out. don’t rush into choosing a framework The framework should be the last thing you worry about, not the first. First, make sure your architecture is solid. Know your core components and how they’ll interact. Once you’ve got that, you can evaluate frameworks with a clear understanding of where they might fit — or if they even fit at all. This approach ensures that your design is solid and suited to your specific needs. When it comes time to consider a framework, you’ll be able to see clearly where it can enhance your architecture without limiting it. Before you jump into using any framework, ask yourself: do you really, truly need it? Sure, frameworks can add layers of automation and convenience, but they also come with their own set of limitations. If your application has unique requirements, frameworks might not play nice with them. do you really, truly need it? Think long and hard about the long-term benefits against the limitations. Making Frameworks Expendable If you decide a framework is worth the risk, make sure it’s easy to replace. Yes, you heard that right. Build in some flexibility so that if you need to ditch it later, it’s not a monumental task. Here’s how: Abstract Your Dependencies Keep the framework’s grubby little hands out of your core code. Use interfaces to abstract the framework’s functionality so that your business logic doesn’t depend on the framework directly. Let’s say you’re using TensorFlow for machine learning. Instead of embedding TensorFlow code throughout your entire application, define interfaces to keep things neat and abstract: from abc import ABC, abstractmethod
import tensorflow as tf

class ModelTrainer(ABC):
    @abstractmethod
    def train(self, data):
        pass

class TensorFlowTrainer(ModelTrainer):
    def train(self, data):
        # TensorFlow-specific training logic
        model = tf.keras.models.Sequential([...])
        model.compile(optimizer='adam', loss='sparse_categorical_crossentropy')
        model.fit(data, epochs=5)
        return model from abc import ABC, abstractmethod
import tensorflow as tf

class ModelTrainer(ABC):
    @abstractmethod
    def train(self, data):
        pass

class TensorFlowTrainer(ModelTrainer):
    def train(self, data):
        # TensorFlow-specific training logic
        model = tf.keras.models.Sequential([...])
        model.compile(optimizer='adam', loss='sparse_categorical_crossentropy')
        model.fit(data, epochs=5)
        return model By doing this, your core logic isn’t tightly coupled with TensorFlow. If you need to switch to another machine learning framework, it’s just a matter of swapping out the implementation. Dependency Injection (DI) is Your Friend Next, let’s talk about Dependency Injection (DI). This technique lets you inject specific implementations of your interfaces into your classes, keeping your codebase decoupled and modular. class TrainingPipeline:
    def __init__(self, trainer: ModelTrainer):
        self.trainer = trainer

    def execute(self, data):
        return self.trainer.train(data)

# Inject the TensorFlowTrainer implementation
pipeline = TrainingPipeline(TensorFlowTrainer()) class TrainingPipeline:
    def __init__(self, trainer: ModelTrainer):
        self.trainer = trainer

    def execute(self, data):
        return self.trainer.train(data)

# Inject the TensorFlowTrainer implementation
pipeline = TrainingPipeline(TensorFlowTrainer()) Now your code is flexible, easy to test, and ready for whatever the future throws at it. Embrace Inversion of Control (IoC) For the ultimate in flexibility, take things up a notch with Inversion of Control (IoC). This pattern allows you to specify implementations in a configuration file or a centralized location in your code. It’s the cherry on top of your framework-agnostic architecture. Here’s an example of how that might work with a configuration-based approach: # config.py
class Config:
    TRAINER = 'my_project.trainers.TensorFlowTrainer'

# main.py
import importlib

class TrainingPipeline:
    def __init__(self, trainer_class: str):
        module_name, class_name = trainer_class.rsplit('.', 1)
        module = importlib.import_module(module_name)
        trainer_cls = getattr(module, class_name)
        self.trainer = trainer_cls()

    def execute(self, data):
        return self.trainer.train(data)

# Inject the trainer specified in the configuration
from config import Config
pipeline = TrainingPipeline(Config.TRAINER) # config.py
class Config:
    TRAINER = 'my_project.trainers.TensorFlowTrainer'

# main.py
import importlib

class TrainingPipeline:
    def __init__(self, trainer_class: str):
        module_name, class_name = trainer_class.rsplit('.', 1)
        module = importlib.import_module(module_name)
        trainer_cls = getattr(module, class_name)
        self.trainer = trainer_cls()

    def execute(self, data):
        return self.trainer.train(data)

# Inject the trainer specified in the configuration
from config import Config
pipeline = TrainingPipeline(Config.TRAINER) Now, if you ever need to replace TensorFlow with another machine learning framework, you simply update the configuration and carry on. No hassle, no drama. Conclusion Remember, frameworks should serve YOUR architecture, not dictate it. With careful planning and strategic abstraction, you can reap the benefits of frameworks without getting trapped in long-term dependencies. The trick is staying in control. So next time you’re about to dive into a framework, take a step back and remind yourself: you call the shots here. Thank you for reading! Thank you for reading! Any questions? Leave your comment below to start fantastic discussions! Check out my blog or come to say hi 👋 on Twitter or subscribe to my telegram channel . Plan your best! my blog Twitter my telegram channel

The Frameworks Dilemma

የ Frameworks አጣብቂኝ

Дылема рамак

El dilema de los marcos

O Dilema dos Frameworks

Dilema okvira

フレームワークのジレンマ

Frameworks dilema

O dilema das estruturas

Дилеммаи Frameworks

I-Frameworks Dilemma

AUSTIN, TX, September 18th, 2024/CyberNewsWire/--Research indicates that an infostealer malware infection is often a precursor to a ransomware attack SpyCloud, the leader in Cybercrime Analytics, today announced new cybersecurity research highlighting the growing and alarming threat of infostealers – a type of malware designed to exfiltrate digital identity data, login credentials, and session cookies from infected devices. SpyCloud’s latest findings reveal the staggering scale of identity exposure caused by infostealers, the influence this type of malware has had on the surge in ransomware incidents, and the profound implications for businesses worldwide. Massive Scale of Identity Exposure Creates New Risks According to SpyCloud, 61% of all data breaches in the past year were malware-related, with infostealers responsible for the theft of 343.78 million credentials. These stolen credentials are then sold in criminal communities for use in further attacks. The research also found that one in five individuals has been a victim of an infostealer infection. Each infection, on average, exposes 10-25 third-party business application credentials, creating fertile ground for further access and exploitation, particularly by ransomware operators. “Our latest findings reveal a critical shift in the cybersecurity landscape,” said Damon Fleury, chief product officer at SpyCloud. “Infostealers have become the go-to tool for cybercriminals, with their ability to exfiltrate valuable data in a matter of seconds, creating a runway for cyberattacks like ransomware off the vast amounts of stolen access to SSO, VPN, admin panels, and other critical applications.” Infostealers: The Precursor To Ransomware Attacks The link between infostealers and ransomware is becoming increasingly evident. Through deep analysis of recaptured infostealer logs, SpyCloud discovered a worrying trend: companies with employees and contractors who are infected with infostealer malware are significantly more likely to experience a ransomware attack. In fact, nearly one-third of companies that suffered a ransomware attack last year had previously experienced an infostealer infection. According to the report, this is based on publicly known incidents and confirmed ransomware events. The true exposure is potentially even higher as not all ransomware incidents are made publicly available. “The correlation between infostealer infections and subsequent ransomware attacks is a wake-up call for businesses,” said Trevor Hilligoss, vice president of SpyCloud Labs, SpyCloud. “However, this field is incredibly complex and fast-moving. This year, we’re seeing new infostealers families that make use of expanded capabilities such as advanced encryption to stay stealthy or the ability to restore expired authentication cookies for more persistent access.” The Rise of Malware-as-a-Service And Account Takeover Attacks The infostealer threat is further exacerbated by the rise of Malware-as-a-Service (MaaS). This off-the-shelf model allows even low-skilled cybercriminals to purchase and deploy sophisticated malware, including infostealers, with ease. Through MaaS, these criminals can acquire fresh and accurate identity data in bulk, fueling the cycle of cybercrime. SpyCloud’s findings also shed light on the evolution of account takeover (ATO) attacks, powered by infostealers. Unlike traditional ATO, which relies on stolen credentials (username and password combinations), next-generation ATO leverages stolen session cookies to sidestep traditional authentication methods in what is known as session hijacking. By taking over these already-authenticated sessions, cybercriminals can mimic legitimate users and infiltrate networks undetected. This method significantly increases the success rate of attacks and poses a severe threat to organizational security. “The sheer volume of credentials and session cookies being siphoned by infostealers is staggering,” said Hilligoss. “In the last 90 days alone, SpyCloud has recaptured over 5.4 billion stolen cookie records – with an average of nearly 2,000 exposed records per infected device. This vast trove of data is increasingly used by ransomware operators and initial access brokers to facilitate their attacks, highlighting the need for advanced defense strategies.” Antivirus, MFA And Traditional Defenses Are No Longer Enough At least 54% of devices infected with infostealers in the first half of 2024 had antivirus or endpoint detection and response (EDR) solutions installed, underscoring the limitations of traditional cybersecurity measures in combating the techniques used by modern cybercriminals. Furthermore, infostealers and session hijacking attacks render multi-factor authentication (MFA) and passwordless authentication methods like passkeys ineffective. By hijacking already-authenticated sessions, cybercriminals can impersonate legitimate users and side-step even the most robust authentication methods. The call for next-generation cybersecurity The findings from SpyCloud make it clear: traditional malware mitigation is no longer sufficient and ignoring the problem only exacerbates the impact on businesses. Organizations must move beyond merely removing infections and focus on remediating the long-term risks posed by exposed data. This includes resetting compromised application credentials and invalidating session cookies siphoned by infostealers. By understanding the risks posed by infostealers and working to mitigate the data that has been exfiltrated, organizations are able to limit the likelihood of devastating cyberattacks such as ransomware that stem from this stolen data. SpyCloud remains committed to helping organizations navigate these challenges and safeguard their digital assets. Readers can download the full 2024 Malware and Ransomware Defense Report. To learn more about how SpyCloud helps organizations defend against ransomware, readers can visit https://spycloud.com/use-case/ransomware-prevention/. About the SpyCloud 2024 Malware and Ransomware Defense Report For this fourth annual report, SpyCloud surveyed 510 individuals in active cybersecurity roles within organizations in the US and the UK with at least 500 employees. The report examines the top concerns and real-life impacts of ransomware, including popular entry points, ransom payments, and the cumulative costs of these attacks to the business. It also highlights key cyber threat prevention strategies and future security priorities identified by these experts. About SpyCloud SpyCloud transforms recaptured darknet data to disrupt cybercrime. Its automated identity threat protection solutions leverage advanced analytics to proactively prevent ransomware and account takeover, safeguard employee and consumer accounts, and accelerate cybercrime investigations. SpyCloud's data from breaches, malware-infected devices, and successful phishes also powers many popular dark web monitoring and identity theft protection offerings. Customers include more than half of the Fortune 10, along with hundreds of global enterprises, mid-sized companies, and government agencies worldwide. Headquartered in Austin, TX, SpyCloud is home to more than 200 cybersecurity experts whose mission is to protect businesses and consumers from the stolen identity data criminals are using to target them now. To learn more and see insights on their company’s exposed data, readers can visit spycloud.com Contact EVP, Public Relations Katie Hanusik REQ on behalf of SpyCloud spycloud@req.co This story was distributed as a release by Cyberwire under HackerNoon’s Business Blogging Program. Learn more about the program here AUSTIN, TX, September 18th, 2024/CyberNewsWire/--Research indicates that an infostealer malware infection is often a precursor to a ransomware attack SpyCloud, the leader in Cybercrime Analytics, today announced new cybersecurity research highlighting the growing and alarming threat of infostealers – a type of malware designed to exfiltrate digital identity data, login credentials, and session cookies from infected devices. SpyCloud’s latest findings reveal the staggering scale of identity exposure caused by infostealers, the influence this type of malware has had on the surge in ransomware incidents, and the profound implications for businesses worldwide. Massive Scale of Identity Exposure Creates New Risks According to SpyCloud, 61% of all data breaches in the past year were malware-related, with infostealers responsible for the theft of 343.78 million credentials. These stolen credentials are then sold in criminal communities for use in further attacks. The research also found that one in five individuals has been a victim of an infostealer infection. Each infection, on average, exposes 10-25 third-party business application credentials, creating fertile ground for further access and exploitation, particularly by ransomware operators. “Our latest findings reveal a critical shift in the cybersecurity landscape,” said Damon Fleury, chief product officer at SpyCloud. “Our latest findings reveal a critical shift in the cybersecurity landscape,” said Damon Fleury, chief product officer at SpyCloud. “Infostealers have become the go-to tool for cybercriminals, with their ability to exfiltrate valuable data in a matter of seconds, creating a runway for cyberattacks like ransomware off the vast amounts of stolen access to SSO, VPN, admin panels, and other critical applications.” “Infostealers have become the go-to tool for cybercriminals, with their ability to exfiltrate valuable data in a matter of seconds, creating a runway for cyberattacks like ransomware off the vast amounts of stolen access to SSO, VPN, admin panels, and other critical applications.” Infostealers: The Precursor To Ransomware Attacks The link between infostealers and ransomware is becoming increasingly evident. Through deep analysis of recaptured infostealer logs, SpyCloud discovered a worrying trend: companies with employees and contractors who are infected with infostealer malware are significantly more likely to experience a ransomware attack. In fact, nearly one-third of companies that suffered a ransomware attack last year had previously experienced an infostealer infection. According to the report, this is based on publicly known incidents and confirmed ransomware events. The true exposure is potentially even higher as not all ransomware incidents are made publicly available. “The correlation between infostealer infections and subsequent ransomware attacks is a wake-up call for businesses,” said Trevor Hilligoss, vice president of SpyCloud Labs, SpyCloud. “The correlation between infostealer infections and subsequent ransomware attacks is a wake-up call for businesses,” said Trevor Hilligoss, vice president of SpyCloud Labs, SpyCloud. “However, this field is incredibly complex and fast-moving. This year, we’re seeing new infostealers families that make use of expanded capabilities such as advanced encryption to stay stealthy or the ability to restore expired authentication cookies for more persistent access.” “However, this field is incredibly complex and fast-moving. This year, we’re seeing new infostealers families that make use of expanded capabilities such as advanced encryption to stay stealthy or the ability to restore expired authentication cookies for more persistent access.” The Rise of Malware-as-a-Service And Account Takeover Attacks The infostealer threat is further exacerbated by the rise of Malware-as-a-Service (MaaS). This off-the-shelf model allows even low-skilled cybercriminals to purchase and deploy sophisticated malware, including infostealers, with ease. Through MaaS, these criminals can acquire fresh and accurate identity data in bulk, fueling the cycle of cybercrime. SpyCloud’s findings also shed light on the evolution of account takeover (ATO) attacks, powered by infostealers. Unlike traditional ATO, which relies on stolen credentials (username and password combinations), next-generation ATO leverages stolen session cookies to sidestep traditional authentication methods in what is known as session hijacking. By taking over these already-authenticated sessions, cybercriminals can mimic legitimate users and infiltrate networks undetected. This method significantly increases the success rate of attacks and poses a severe threat to organizational security. “The sheer volume of credentials and session cookies being siphoned by infostealers is staggering,” said Hilligoss. “The sheer volume of credentials and session cookies being siphoned by infostealers is staggering,” said Hilligoss. “In the last 90 days alone, SpyCloud has recaptured over 5.4 billion stolen cookie records – with an average of nearly 2,000 exposed records per infected device. This vast trove of data is increasingly used by ransomware operators and initial access brokers to facilitate their attacks, highlighting the need for advanced defense strategies.” “In the last 90 days alone, SpyCloud has recaptured over 5.4 billion stolen cookie records – with an average of nearly 2,000 exposed records per infected device. This vast trove of data is increasingly used by ransomware operators and initial access brokers to facilitate their attacks, highlighting the need for advanced defense strategies.” Antivirus, MFA And Traditional Defenses Are No Longer Enough At least 54% of devices infected with infostealers in the first half of 2024 had antivirus or endpoint detection and response (EDR) solutions installed, underscoring the limitations of traditional cybersecurity measures in combating the techniques used by modern cybercriminals. Furthermore, infostealers and session hijacking attacks render multi-factor authentication (MFA) and passwordless authentication methods like passkeys ineffective. By hijacking already-authenticated sessions, cybercriminals can impersonate legitimate users and side-step even the most robust authentication methods. The call for next-generation cybersecurity The findings from SpyCloud make it clear: traditional malware mitigation is no longer sufficient and ignoring the problem only exacerbates the impact on businesses. Organizations must move beyond merely removing infections and focus on remediating the long-term risks posed by exposed data. This includes resetting compromised application credentials and invalidating session cookies siphoned by infostealers. By understanding the risks posed by infostealers and working to mitigate the data that has been exfiltrated, organizations are able to limit the likelihood of devastating cyberattacks such as ransomware that stem from this stolen data. SpyCloud remains committed to helping organizations navigate these challenges and safeguard their digital assets. Readers can download the full 2024 Malware and Ransomware Defense Report . 2024 Malware and Ransomware Defense Report 2024 Malware and Ransomware Defense Report To learn more about how SpyCloud helps organizations defend against ransomware, readers can visit https://spycloud.com/use-case/ransomware-prevention/ . https://spycloud.com/use-case/ransomware-prevention/ https://spycloud.com/use-case/ransomware-prevention/ About the SpyCloud 2024 Malware and Ransomware Defense Report For this fourth annual report , SpyCloud surveyed 510 individuals in active cybersecurity roles within organizations in the US and the UK with at least 500 employees. fourth annual report fourth annual report The report examines the top concerns and real-life impacts of ransomware, including popular entry points, ransom payments, and the cumulative costs of these attacks to the business. It also highlights key cyber threat prevention strategies and future security priorities identified by these experts. About SpyCloud SpyCloud transforms recaptured darknet data to disrupt cybercrime. Its automated identity threat protection solutions leverage advanced analytics to proactively prevent ransomware and account takeover, safeguard employee and consumer accounts, and accelerate cybercrime investigations. SpyCloud SpyCloud SpyCloud's data from breaches, malware-infected devices, and successful phishes also powers many popular dark web monitoring and identity theft protection offerings. Customers include more than half of the Fortune 10, along with hundreds of global enterprises, mid-sized companies, and government agencies worldwide. Headquartered in Austin, TX, SpyCloud is home to more than 200 cybersecurity experts whose mission is to protect businesses and consumers from the stolen identity data criminals are using to target them now. To learn more and see insights on their company’s exposed data, readers can visit spycloud.com spycloud.com spycloud.com Contact EVP, Public Relations Katie Hanusik REQ on behalf of SpyCloud spycloud@req.co This story was distributed as a release by Cyberwire under HackerNoon’s Business Blogging Program. Learn more about the program here This story was distributed as a release by Cyberwire under HackerNoon’s Business Blogging Program. Learn more about the program here here

SpyCloud Unveils Massive Scale Of Identity Exposure, Highlighting Need For Advanced Security Measure

SpyCloud تكشف عن نطاق هائل من الكشف عن الهوية، مما يسلط الضوء على الحاجة إلى تدابير أمنية متقدمة

SpyCloud ukax Jach’a Escala de Exposición de Identidad ukar uñacht’ayi, Medida de Seguridad Avanzada ukan munatapat uñacht’ayi

SpyCloud revela una escala masiva de exposición de identidad, lo que pone de relieve la necesidad de medidas de seguridad avanzadas

SpyCloud مقیاس عظیمی از قرار گرفتن در معرض هویت را معرفی می کند، که نیاز به اقدامات امنیتی پیشرفته را برجسته می کند

स्पाईक्लाउड ने पहचान उजागर करने के बड़े पैमाने का खुलासा किया, उन्नत सुरक्षा उपाय की आवश्यकता पर प्रकाश डाला

SpyCloud が大規模な個人情報漏洩を発表、高度なセキュリティ対策の必要性を浮き彫りに

SpyCloud გამოაქვეყნებს პირადობის ექსპოზიციის მასიურ მასშტაბებს, ხაზს უსვამს უსაფრთხოების გაფართოებული ზომების საჭიროებას

SpyCloud විසින් අනන්‍යතා නිරාවරණයේ දැවැන්ත පරිමාණයක් එළිදක්වයි, උසස් ආරක්ෂක පියවරක අවශ්‍යතාවය ඉස්මතු කරයි

СпиЦлоуд открива огромну скалу откривања идентитета, наглашавајући потребу за напредном мјером сигурности

SpyCloud Inafichua Kiwango Kikubwa cha Ufichuaji wa Utambulisho, Ikiangazia Haja ya Kipimo cha Kina Usalama

ஸ்பைக்ளவுட் அடையாள வெளிப்பாட்டின் பாரிய அளவை வெளிப்படுத்துகிறது, மேம்பட்ட பாதுகாப்பு நடவடிக்கையின் தேவையை எடுத்துக்காட்டுகிறது

SpyCloud tiết lộ quy mô lớn về việc tiết lộ danh tính, làm nổi bật nhu cầu về biện pháp bảo mật tiên tiến

Building a Device Manager App on Android: Lessons from Managing 300K+ Terminals

Finally, ELIZA: A Platform, Not a ChatBot!

Stories, News & Mentions about THESHELF

Theshelf’s Stories on HackerNoon