Baselime

baselime

Authors:
(1) Rafael Rafailo, Stanford University and Equal contribution; more junior authors listed earlier;
(2) Archit Sharma, Stanford University and Equal contribution; more junior authors listed earlier;
(3) Eric Mitchel, Stanford University and Equal contribution; more junior authors listed earlier;
(4) Stefano Ermon, CZ Biohub;
(5) Christopher D. Manning, Stanford University;
(6) Chelsea Finn, Stanford University. Table of Links Abstract and 1. Introduction 2 Related Work 3 Preliminaries 4 Direct Preference Optimization 5 Theoretical Analysis of DPO 6 Experiments 7 Discussion, Acknowledgements, and References Author Contributions A Mathematical Derivations A.1 Deriving the Optimum of the KL-Constrained Reward Maximization Objective A.2 Deriving the DPO Objective Under the Bradley-Terry Model A.3 Deriving the DPO Objective Under the Plackett-Luce Model A.4 Deriving the Gradient of the DPO Objective and A.5 Proof of Lemma 1 and 2 A.6 Proof of Theorem 1 B DPO Implementation Details and Hyperparameters C Further Details on the Experimental Set-Up and C.1 IMDb Sentiment Experiment and Baseline Details C.2 GPT-4 prompts for computing summarization and dialogue win rates C.3 Unlikelihood baseline D Additional Empirical Results D.1 Performance of Best of N baseline for Various N and D.2 Sample Responses and GPT-4 Judgments D.3 Human study details Additional Empirical Results D.1 Performance of Best of N baseline for Various N We find that the Best of N baseline is a strong (although computationally expensive, requiring sampling many times) baseline in our experiments. We include an evaluation of the Best of N baseline for various N for the Anthropic-HH dialogue and TL;DR summarization; the results are shown in Figure 4. D.2 Sample Responses and GPT-4 Judgments In this section, we present examples of comparisons between DPO and the baseline (PPO temp 0. for summarization, and the ground truth chosen response for dialogue). See Tables 4-6 for summarization examples, and Tables 7-10 for dialogue examples. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Rafael Rafailo, Stanford University and Equal contribution; more junior authors listed earlier; (2) Archit Sharma, Stanford University and Equal contribution; more junior authors listed earlier; (3) Eric Mitchel, Stanford University and Equal contribution; more junior authors listed earlier; (4) Stefano Ermon, CZ Biohub; (5) Christopher D. Manning, Stanford University; (6) Chelsea Finn, Stanford University. Authors: Authors: (1) Rafael Rafailo, Stanford University and Equal contribution; more junior authors listed earlier; (2) Archit Sharma, Stanford University and Equal contribution; more junior authors listed earlier; (3) Eric Mitchel, Stanford University and Equal contribution; more junior authors listed earlier; (4) Stefano Ermon, CZ Biohub; (5) Christopher D. Manning, Stanford University; (6) Chelsea Finn, Stanford University. Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2 Related Work 2 Related Work 3 Preliminaries 3 Preliminaries 4 Direct Preference Optimization 4 Direct Preference Optimization 5 Theoretical Analysis of DPO 5 Theoretical Analysis of DPO 6 Experiments 6 Experiments 7 Discussion, Acknowledgements, and References 7 Discussion, Acknowledgements, and References Author Contributions Author Contributions A Mathematical Derivations A.1 Deriving the Optimum of the KL-Constrained Reward Maximization Objective A.1 Deriving the Optimum of the KL-Constrained Reward Maximization Objective A.2 Deriving the DPO Objective Under the Bradley-Terry Model A.2 Deriving the DPO Objective Under the Bradley-Terry Model A.3 Deriving the DPO Objective Under the Plackett-Luce Model A.3 Deriving the DPO Objective Under the Plackett-Luce Model A.4 Deriving the Gradient of the DPO Objective and A.5 Proof of Lemma 1 and 2 A.4 Deriving the Gradient of the DPO Objective and A.5 Proof of Lemma 1 and 2 A.6 Proof of Theorem 1 A.6 Proof of Theorem 1 B DPO Implementation Details and Hyperparameters B DPO Implementation Details and Hyperparameters C Further Details on the Experimental Set-Up and C.1 IMDb Sentiment Experiment and Baseline Details C Further Details on the Experimental Set-Up and C.1 IMDb Sentiment Experiment and Baseline Details C.2 GPT-4 prompts for computing summarization and dialogue win rates C.2 GPT-4 prompts for computing summarization and dialogue win rates C.3 Unlikelihood baseline C.3 Unlikelihood baseline D Additional Empirical Results D.1 Performance of Best of N baseline for Various N and D.2 Sample Responses and GPT-4 Judgments D.1 Performance of Best of N baseline for Various N and D.2 Sample Responses and GPT-4 Judgments D.3 Human study details D.3 Human study details Additional Empirical Results D.1 Performance of Best of N baseline for Various N We find that the Best of N baseline is a strong (although computationally expensive, requiring sampling many times) baseline in our experiments. We include an evaluation of the Best of N baseline for various N for the Anthropic-HH dialogue and TL;DR summarization; the results are shown in Figure 4. D.2 Sample Responses and GPT-4 Judgments In this section, we present examples of comparisons between DPO and the baseline (PPO temp 0. for summarization, and the ground truth chosen response for dialogue). See Tables 4-6 for summarization examples, and Tables 7-10 for dialogue examples. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Performance of Best of N Baseline for Various N and Sample Responses and GPT-4 Judgments

Authors:
(1) Rafael Rafailo, Stanford University and Equal contribution; more junior authors listed earlier;
(2) Archit Sharma, Stanford University and Equal contribution; more junior authors listed earlier;
(3) Eric Mitchel, Stanford University and Equal contribution; more junior authors listed earlier;
(4) Stefano Ermon, CZ Biohub;
(5) Christopher D. Manning, Stanford University;
(6) Chelsea Finn, Stanford University. Table of Links Abstract and 1. Introduction 2 Related Work 3 Preliminaries 4 Direct Preference Optimization 5 Theoretical Analysis of DPO 6 Experiments 7 Discussion, Acknowledgements, and References Author Contributions A Mathematical Derivations A.1 Deriving the Optimum of the KL-Constrained Reward Maximization Objective A.2 Deriving the DPO Objective Under the Bradley-Terry Model A.3 Deriving the DPO Objective Under the Plackett-Luce Model A.4 Deriving the Gradient of the DPO Objective and A.5 Proof of Lemma 1 and 2 A.6 Proof of Theorem 1 B DPO Implementation Details and Hyperparameters C Further Details on the Experimental Set-Up and C.1 IMDb Sentiment Experiment and Baseline Details C.2 GPT-4 prompts for computing summarization and dialogue win rates C.3 Unlikelihood baseline D Additional Empirical Results D.1 Performance of Best of N baseline for Various N and D.2 Sample Responses and GPT-4 Judgments D.3 Human study details C.3 Unlikelihood baseline This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Rafael Rafailo, Stanford University and Equal contribution; more junior authors listed earlier; (2) Archit Sharma, Stanford University and Equal contribution; more junior authors listed earlier; (3) Eric Mitchel, Stanford University and Equal contribution; more junior authors listed earlier; (4) Stefano Ermon, CZ Biohub; (5) Christopher D. Manning, Stanford University; (6) Chelsea Finn, Stanford University. Authors: Authors: (1) Rafael Rafailo, Stanford University and Equal contribution; more junior authors listed earlier; (2) Archit Sharma, Stanford University and Equal contribution; more junior authors listed earlier; (3) Eric Mitchel, Stanford University and Equal contribution; more junior authors listed earlier; (4) Stefano Ermon, CZ Biohub; (5) Christopher D. Manning, Stanford University; (6) Chelsea Finn, Stanford University. Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2 Related Work 2 Related Work 3 Preliminaries 3 Preliminaries 4 Direct Preference Optimization 4 Direct Preference Optimization 5 Theoretical Analysis of DPO 5 Theoretical Analysis of DPO 6 Experiments 6 Experiments 7 Discussion, Acknowledgements, and References 7 Discussion, Acknowledgements, and References Author Contributions Author Contributions A Mathematical Derivations A.1 Deriving the Optimum of the KL-Constrained Reward Maximization Objective A.1 Deriving the Optimum of the KL-Constrained Reward Maximization Objective A.2 Deriving the DPO Objective Under the Bradley-Terry Model A.2 Deriving the DPO Objective Under the Bradley-Terry Model A.3 Deriving the DPO Objective Under the Plackett-Luce Model A.3 Deriving the DPO Objective Under the Plackett-Luce Model A.4 Deriving the Gradient of the DPO Objective and A.5 Proof of Lemma 1 and 2 A.4 Deriving the Gradient of the DPO Objective and A.5 Proof of Lemma 1 and 2 A.6 Proof of Theorem 1 A.6 Proof of Theorem 1 B DPO Implementation Details and Hyperparameters B DPO Implementation Details and Hyperparameters C Further Details on the Experimental Set-Up and C.1 IMDb Sentiment Experiment and Baseline Details C Further Details on the Experimental Set-Up and C.1 IMDb Sentiment Experiment and Baseline Details C.2 GPT-4 prompts for computing summarization and dialogue win rates C.2 GPT-4 prompts for computing summarization and dialogue win rates C.3 Unlikelihood baseline C.3 Unlikelihood baseline D Additional Empirical Results D.1 Performance of Best of N baseline for Various N and D.2 Sample Responses and GPT-4 Judgments D.1 Performance of Best of N baseline for Various N and D.2 Sample Responses and GPT-4 Judgments D.3 Human study details D.3 Human study details C.3 Unlikelihood baseline This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

The Unlikelihood Baseline in Sentiment Experiments

Authors:
(1) Xiao-Yang Liu, Hongyang Yang, Columbia University (xl2427,hy2500@columbia.edu);
(2) Jiechao Gao, University of Virginia (jg5ycn@virginia.edu);
(3) Christina Dan Wang (Corresponding Author), New York University Shanghai (christina.wang@nyu.edu). Table of Links Abstract and 1 Introduction 2 Related Works and 2.1 Deep Reinforcement Learning Algorithms 2.2 Deep Reinforcement Learning Libraries and 2.3 Deep Reinforcement Learning in Finance 3 The Proposed FinRL Framework and 3.1 Overview of FinRL Framework 3.2 Application Layer 3.3 Agent Layer 3.4 Environment Layer 3.5 Training-Testing-Trading Pipeline 4 Hands-on Tutorials and Benchmark Performance and 4.1 Backtesting Module 4.2 Baseline Strategies and Trading Metrics 4.3 Hands-on Tutorials 4.4 Use Case I: Stock Trading 4.5 Use Case II: Portfolio Allocation and 4.6 Use Case III: Cryptocurrencies Trading 5 Ecosystem of FinRL and Conclusions, and References 4.2 Baseline Strategies and Trading Metrics Baseline trading strategies are provided to compare with DRL strategies. Investors usually have two mutually conflicting objectives: the highest possible profits and the lowest possible risks [43]. We include three conventional strategies as baselines. Passive trading strategy [31] is an easy and popular strategy that has the minimal trading activities. Investors simply buy and hold index ETFs [46] to replicate a broad market index or indices such as Dow Jones Industrial Average (DJIA) index and Standard & Poor’s 500 (S&P 500) index. Mean-variance and min-variance strategy [2] both aim to achieve an optimal balance between the risks and profits. It selects a diversified portfolio with risky assets, and the risk is diversified when traded together. Equally weighted strategy is a type of portfolio allocation method. It gives the same importance to each asset in a portfolio. FinRL includes common metrics to evaluate trading performance: Final portfolio value: the amount of money at the end of the trading period. Cumulative return: subtracting the initial value from the final portfolio value, then dividing by the initial value. Annualized return and standard deviation: geometric average return in a yearly sense, and the corresponding deviation. Maximum drawdown ratio: the maximum observed loss from a historical peak to a trough of a portfolio, before a new peak is achieved. Maximum drawdown is an indicator of downside risk over a time period. Sharpe ratio in (1) is the average return earned in excess of the risk-free rate per unit of volatility. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Xiao-Yang Liu, Hongyang Yang, Columbia University (xl2427,hy2500@columbia.edu); (2) Jiechao Gao, University of Virginia (jg5ycn@virginia.edu); (3) Christina Dan Wang (Corresponding Author), New York University Shanghai (christina.wang@nyu.edu). Authors: Authors: (1) Xiao-Yang Liu, Hongyang Yang, Columbia University (xl2427,hy2500@columbia.edu); (2) Jiechao Gao, University of Virginia (jg5ycn@virginia.edu); (3) Christina Dan Wang ( Corresponding Author ), New York University Shanghai (christina.wang@nyu.edu). Corresponding Author Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Related Works and 2.1 Deep Reinforcement Learning Algorithms 2 Related Works and 2.1 Deep Reinforcement Learning Algorithms 2.2 Deep Reinforcement Learning Libraries and 2.3 Deep Reinforcement Learning in Finance 2.2 Deep Reinforcement Learning Libraries and 2.3 Deep Reinforcement Learning in Finance 3 The Proposed FinRL Framework and 3.1 Overview of FinRL Framework 3 The Proposed FinRL Framework and 3.1 Overview of FinRL Framework 3.2 Application Layer 3.2 Application Layer 3.3 Agent Layer 3.3 Agent Layer 3.4 Environment Layer 3.4 Environment Layer 3.5 Training-Testing-Trading Pipeline 3.5 Training-Testing-Trading Pipeline 4 Hands-on Tutorials and Benchmark Performance and 4.1 Backtesting Module 4 Hands-on Tutorials and Benchmark Performance and 4.1 Backtesting Module 4.2 Baseline Strategies and Trading Metrics 4.2 Baseline Strategies and Trading Metrics 4.3 Hands-on Tutorials 4.3 Hands-on Tutorials 4.4 Use Case I: Stock Trading 4.4 Use Case I: Stock Trading 4.5 Use Case II: Portfolio Allocation and 4.6 Use Case III: Cryptocurrencies Trading 4.5 Use Case II: Portfolio Allocation and 4.6 Use Case III: Cryptocurrencies Trading 5 Ecosystem of FinRL and Conclusions, and References 5 Ecosystem of FinRL and Conclusions, and References 4.2 Baseline Strategies and Trading Metrics Baseline trading strategies are provided to compare with DRL strategies. Investors usually have two mutually conflicting objectives: the highest possible profits and the lowest possible risks [43]. We include three conventional strategies as baselines. Passive trading strategy [31] is an easy and popular strategy that has the minimal trading activities. Investors simply buy and hold index ETFs [46] to replicate a broad market index or indices such as Dow Jones Industrial Average (DJIA) index and Standard & Poor’s 500 (S&P 500) index. Passive trading strategy Mean-variance and min-variance strategy [2] both aim to achieve an optimal balance between the risks and profits. It selects a diversified portfolio with risky assets, and the risk is diversified when traded together. Mean-variance and min-variance strategy Equally weighted strategy is a type of portfolio allocation method. It gives the same importance to each asset in a portfolio. Equally weighted strategy FinRL includes common metrics to evaluate trading performance: Final portfolio value : the amount of money at the end of the trading period. Final portfolio value Cumulative return : subtracting the initial value from the final portfolio value, then dividing by the initial value. Cumulative return Annualized return and standard deviation : geometric average return in a yearly sense, and the corresponding deviation. Annualized return and standard deviation Maximum drawdown ratio : the maximum observed loss from a historical peak to a trough of a portfolio, before a new peak is achieved. Maximum drawdown is an indicator of downside risk over a time period. Maximum drawdown ratio Sharpe ratio in (1) is the average return earned in excess of the risk-free rate per unit of volatility. Sharpe ratio This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Unveiling FinRL's Baseline Strategies and Key Trading Metrics for Portfolio Evaluation

For years, web developers of all experience levels have relied on __caniuse.com __to understand the support of a given feature. Reviewing caniuse.com is helpful in determining backward compatibility or which of the modern browsers supports cutting-edge improvements to web specifications. In 2023, Google introduced Baseline to the web community. This tool is not just another addition to the developer's toolkit but a crucial aid in determining the safety of features and APIs for use in production-grade web applications. It’s a platform for interoperability, a much-needed resource to help engineers produce more consistent and efficient experiences, and a testament to Google's commitment to improving web best practices. Why Google Baseline? Browsers are constantly evolving, and vendors release updates monthly. With more individuals contributing and working on specifications than ever before and a far more transparent dialogue between the main browser vendors (Safari, Firefox, Chrome, Edge), it's become increasingly difficult to track which new features are stable across the landscape. However, with the introduction of Baseline, this struggle is significantly alleviated. Considering the above, as new features are released into browser engines, engineers often need more time to ramp up, refactor, or strategically plan the introduction of new features into their code bases. More importantly, as Google is often one of the platforms that push front-end technology forward, it has admittedly accepted that it is partly to blame here. It’s one of the reasons the Chrome team came up with web.dev and developer.chrome.com, as Chrome is often the first browser to land cutting-edge technologies. The introduction of Baseline has also led to further documentation and conferences focused on interoperability. New features mean documentation, and Google has made strides to ensure MDN has the relevant documentation for newer technologies as they land in Chrome. Closing the Vendor Gap From Google's perspective, engineers have one significant struggle: the behaviour and consistency of features across vendors and browser engines. It’s a pain point for all engineers in the field, requiring complex planning and solutions often to overcome a particular browser engine's lack of support. The solution here (for the last two years) was Interop 2022 and Interop 2023 - a conference and community that helps bridge the gap between browsers to provide a more consistent and performant web. What is Baseline? Baseline serves as the universal flag of acceptance across browser vendors. It helps engineers determine what is fully stable across the landscape and across browser vendors for the most recent two versions of each browser. This feature stability assurance is a game-changer in web development, providing a reliable benchmark for engineers to base their decisions on. How Does Baseline Work? You can access Google Baseline on MDN, web.dev and caniuse.com. Baseline also plans to provide a way to see what has become stable across all browser vendors through a year-on-year approach. For instance, you can read the following article to see what got added to Baseline in 2023: Baseline 2023 As engineering teams, we must keep up with these changes and avoid getting stuck in old solutions for common problems. Importantly, Baseline has two labels: Newly available: the feature is finally interoperable.
Widely available: 30 months have passed since the interoperable date. Baseline considers the major browser vendors to be: Safari (macOS and iOS)
Firefox (Desktop + Android)
Chrome (Desktop + Android)
Edge (Desktop) Google Baseline’s Effect on Engineering Teams One of the most significant benefits of Google Baseline for engineering teams is the removal of guesswork and research required to implement new features. For instance, a Frontend Engineer may want to leverage CSS Grid's new sub-grid feature. With Baseline, they can quickly and confidently determine its support across major browsers, saving time and effort. Sub-grid helps engineers define a grid within a grid to have more refined control over layouts. Heading to caniuse.com tells us there’s a global usage percentage of 87.42%—pretty decent, but we still need to ascertain which browser vendors support / don’t support / partially support the feature. CanIUse now displays a Baseline badge that tells us that this feature is “Newly available across major browsers.” Likewise, if you review the documentation for the sub-grid on MDN, you’ll see a similar banner: Finally, if you review the sub-grid article on web.dev, you’ll see this feature got added to Baseline in 2023: A Word on Evergreen Browsers In my opinion, we do not talk enough about this topic. For most modern users, the days of manually updating browsers are gone. Chrome, Firefox, and Edge auto-update automatically. Safari is an outlier in this case. Safari requires a manual update when a new release is sent out from Apple or when OS X is upgraded annually. Regardless, more users than ever use the latest version of browsers. Although we should ensure the features we build degrade gracefully, this is less of a fundamental consideration nowadays and more of a luxury assumption. Useful Links Google Baseline
Google Baseline Introduction
Baseline 2023
Interop 2022 / Interop 2023 For years, web developers of all experience levels have relied on __ caniuse.com __to understand the support of a given feature. Reviewing caniuse.com is helpful in determining backward compatibility or which of the modern browsers supports cutting-edge improvements to web specifications. caniuse.com In 2023, Google introduced Baseline to the web community. This tool is not just another addition to the developer's toolkit but a crucial aid in determining the safety of features and APIs for use in production-grade web applications. It’s a platform for interoperability, a much-needed resource to help engineers produce more consistent and efficient experiences, and a testament to Google's commitment to improving web best practices. Baseline Baseline Why Google Baseline? Browsers are constantly evolving, and vendors release updates monthly. With more individuals contributing and working on specifications than ever before and a far more transparent dialogue between the main browser vendors (Safari, Firefox, Chrome, Edge), it's become increasingly difficult to track which new features are stable across the landscape. However, with the introduction of Baseline, this struggle is significantly alleviated. Considering the above, as new features are released into browser engines, engineers often need more time to ramp up, refactor, or strategically plan the introduction of new features into their code bases. More importantly, as Google is often one of the platforms that push front-end technology forward, it has admittedly accepted that it is partly to blame here. It’s one of the reasons the Chrome team came up with web.dev and developer.chrome.com , as Chrome is often the first browser to land cutting-edge technologies. The introduction of Baseline has also led to further documentation and conferences focused on interoperability. web.dev web.dev developer.chrome.com developer.chrome.com New features mean documentation, and Google has made strides to ensure MDN has the relevant documentation for newer technologies as they land in Chrome. Closing the Vendor Gap From Google's perspective, engineers have one significant struggle: the behaviour and consistency of features across vendors and browser engines . It’s a pain point for all engineers in the field, requiring complex planning and solutions often to overcome a particular browser engine's lack of support. the behaviour and consistency of features across vendors and browser engines The solution here (for the last two years) was Interop 2022 and Interop 2023 - a conference and community that helps bridge the gap between browsers to provide a more consistent and performant web. Interop 2022 Interop 2022 Interop 2023 Interop 2023 What is Baseline? Baseline serves as the universal flag of acceptance across browser vendors. It helps engineers determine what is fully stable across the landscape and across browser vendors for the most recent two versions of each browser. This feature stability assurance is a game-changer in web development, providing a reliable benchmark for engineers to base their decisions on. How Does Baseline Work? You can access Google Baseline on MDN , web.dev and caniuse.com . MDN MDN web.dev web.dev caniuse.com caniuse.com Baseline also plans to provide a way to see what has become stable across all browser vendors through a year-on-year approach. For instance, you can read the following article to see what got added to Baseline in 2023: Baseline 2023 Baseline 2023 Baseline 2023 As engineering teams, we must keep up with these changes and avoid getting stuck in old solutions for common problems. Importantly, Baseline has two labels: Newly available: the feature is finally interoperable. Widely available: 30 months have passed since the interoperable date. Newly available: the feature is finally interoperable. Widely available: 30 months have passed since the interoperable date. Baseline considers the major browser vendors to be: Safari (macOS and iOS) Firefox (Desktop + Android) Chrome (Desktop + Android) Edge (Desktop) Safari (macOS and iOS) Safari Safari Firefox (Desktop + Android) Firefox Firefox Chrome (Desktop + Android) Chrome Chrome Edge (Desktop) Edge Edge Google Baseline’s Effect on Engineering Teams One of the most significant benefits of Google Baseline for engineering teams is the removal of guesswork and research required to implement new features. For instance, a Frontend Engineer may want to leverage CSS Grid's new sub-grid feature. With Baseline, they can quickly and confidently determine its support across major browsers, saving time and effort. sub-grid sub-grid Sub-grid helps engineers define a grid within a grid to have more refined control over layouts. Heading to caniuse.com tells us there’s a global usage percentage of 87.42%—pretty decent, but we still need to ascertain which browser vendors support / don’t support / partially support the feature. caniuse.com caniuse.com CanIUse now displays a Baseline badge that tells us that this feature is “ Newly available across major browsers. ” CanIUse CanIUse Newly available across major browsers. Likewise, if you review the documentation for the sub-grid on MDN, you’ll see a similar banner: Finally, if you review the sub-grid article on web.dev , you’ll see this feature got added to Baseline in 2023 : web.dev web.dev Baseline in 2023 Baseline in 2023 A Word on Evergreen Browsers In my opinion, we do not talk enough about this topic. For most modern users, the days of manually updating browsers are gone. Chrome, Firefox, and Edge auto-update automatically. Safari is an outlier in this case. Safari requires a manual update when a new release is sent out from Apple or when OS X is upgraded annually. Regardless, more users than ever use the latest version of browsers. Although we should ensure the features we build degrade gracefully, this is less of a fundamental consideration nowadays and more of a luxury assumption. Useful Links Google Baseline Google Baseline Introduction Baseline 2023 Interop 2022 / Interop 2023 Google Baseline Google Baseline Google Baseline Google Baseline Introduction Google Baseline Introduction Google Baseline Introduction Baseline 2023 Baseline 2023 Baseline 2023 Interop 2022 / Interop 2023 Interop 2022 Interop 2022 Interop 2023 Interop 2023

What Is Google Baseline?: A Quick Look at Its Impact on Browser Compatibility and Interoperability

Starting Simple: The Strategic Advantage of Baseline Models in Machine Learning

Begin eenvoudig: die strategiese voordeel van basislynmodelle in masjienleer

ቀላል ጅምር፡ በማሽን መማሪያ ውስጥ የመነሻ ሞዴሎች ስልታዊ ጥቅም

البدء ببساطة: الميزة الاستراتيجية للنماذج الأساسية في التعلم الآلي

Sapuru qalltaña: Ventaja Estratégica de Modelos de Línea Base en Aprendizaje Máquina

Sadə Başlanğıc: Maşın Öyrənməsində Baza Modellərinin Strateji Üstünlüyü

Пачаць з простага: стратэгічная перавага базавых мадэляў у машынным навучанні

Да започнем лесно: Стратегическото предимство на базовите модели в машинното обучение

Počevši od jednostavnog: Strateška prednost osnovnih modela u mašinskom učenju

Començant senzill: l'avantatge estratègic dels models de referència en l'aprenentatge automàtic

Začít jednoduše: Strategická výhoda základních modelů ve strojovém učení

Begyndende enkelt: Den strategiske fordel ved basismodeller i maskinlæring

Ξεκινώντας απλά: Το στρατηγικό πλεονέκτημα των βασικών μοντέλων στη μηχανική μάθηση

Hasi sinplea: oinarrizko ereduen abantaila estrategikoa ikaskuntza automatikoan

شروع ساده: مزیت استراتژیک مدل های پایه در یادگیری ماشین

Aloitetaan yksinkertaisesta: perusmallien strateginen etu koneoppimisessa

Comezando sinxelo: a vantaxe estratéxica dos modelos de referencia na aprendizaxe automática

התחלה פשוטה: היתרון האסטרטגי של מודלים בסיסיים בלמידת מכונה

Početi jednostavno: strateška prednost osnovnih modela u strojnom učenju

Kòmanse Senp: Avantaj Estratejik nan Modèl debaz nan aprantisaj machin

Kezdve egyszerűen: Az alapmodellek stratégiai előnyei a gépi tanulásban

Սկսած պարզ. Հիմնական մոդելների ռազմավարական առավելությունը մեքենայական ուսուցման մեջ

Memulai dengan Sederhana: Keunggulan Strategis Model Dasar dalam Pembelajaran Mesin

Iniziare in modo semplice: il vantaggio strategico dei modelli di base nell'apprendimento automatico

საწყისი მარტივია: საბაზისო მოდელების სტრატეგიული უპირატესობა მანქანათმცოდნეობაში

Қарапайымнан бастау: машиналық оқытудағы базалық үлгілердің стратегиялық артықшылығы

ការចាប់ផ្តើមសាមញ្ញ៖ អត្ថប្រយោជន៍ជាយុទ្ធសាស្រ្តនៃគំរូមូលដ្ឋានក្នុងការរៀនម៉ាស៊ីន

Жөнөкөйдөн баштоо: Машиналарды үйрөнүүдө базалык моделдердин стратегиялык артыкчылыгы

Kobanda na pete: Avantage stratégique ya ba modèles ya base na apprentissage automatique

ເລີ່ມຕົ້ນງ່າຍດາຍ: ຄວາມໄດ້ປຽບຍຸດທະສາດຂອງຕົວແບບພື້ນຖານໃນການຮຽນຮູ້ເຄື່ອງຈັກ

Pradedant paprastai: strateginis bazinių modelių pranašumas mašininio mokymosi srityje

Sākot ar vienkāršu: bāzes modeļu stratēģiskās priekšrocības mašīnmācībā

Manomboka tsotra: Ny tombony stratejika amin'ny maodely fototra amin'ny fianarana milina

Едноставно започнување: Стратешката предност на основните модели во машинското учење

Энгийн эхлэл: Машины сургалтын үндсэн загваруудын стратегийн давуу тал

Bermula Mudah: Kelebihan Strategik Model Baseline dalam Pembelajaran Mesin

सुरु गर्ने सरल: मेसिन लर्निङमा आधारभूत मोडेलहरूको रणनीतिक लाभ

Eenvoudig beginnen: het strategische voordeel van basismodellen in machinaal leren

Go Thoma ka Bonolo: Mohola wa Togamaano wa Mehlala ya Motheo go Thutong ya Motšhene

Zaczynając od prostoty: strategiczna przewaga modeli bazowych w uczeniu maszynowym

ساده پیل کول: د ماشین زده کړې کې د اساسی ماډلونو ستراتیژیک ګټه

Simple qallariy: Ventaja Estratégica nisqa Modelos de Base nisqamanta Maquina Aprendizaje nisqapi

Începe simplu: avantajul strategic al modelelor de bază în învățarea automată

Gutangira Byoroshye: Ibyiza Byingirakamaro Byibanze Byibanze mu Kwiga Imashini

සරලව ආරම්භ කිරීම: යන්ත්‍ර ඉගෙනීමේ මූලික මාදිලිවල උපාය මාර්ගික වාසිය

Začať jednoducho: Strategická výhoda základných modelov v strojovom učení

Kutanga Nyore: Iyo Strategic Advantage yeBaseline Models muMuchina Kudzidza

Bilaabida Fudud: Faa'iidada Istaraatiijiyadeed ee Moodooyinka Saldhigga ee Barashada Mashiinka

Fillimi i thjeshtë: Avantazhi Strategjik i Modeleve Bazë në Mësimin e Makinerisë

Почевши од једноставног: стратешка предност основних модела у машинском учењу

Att börja enkelt: Den strategiska fördelen med basmodeller i maskininlärning

Kuanza Rahisi: Manufaa ya Kimkakati ya Miundo ya Msingi katika Kujifunza kwa Mashine

எளிமையாக தொடங்குதல்: இயந்திர கற்றலில் அடிப்படை மாதிரிகளின் மூலோபாய நன்மை

Оғози оддӣ: Бартарии стратегии моделҳои ибтидоӣ дар омӯзиши мошин

เริ่มต้นง่ายๆ: ข้อได้เปรียบเชิงกลยุทธ์ของโมเดลพื้นฐานในการเรียนรู้ของเครื่องจักร

ቀሊል ምጅማር፡ ስትራተጂካዊ ብልጫ መሰረታዊ ሞዴላት ኣብ ማሽን ትምህርቲ

Pleönekeýje başlamak: Maşyn öwrenmekde esasy modelleriň strategiki artykmaçlygy

Simpleng Simula: Ang Madiskarteng Advantage ng Baseline Models sa Machine Learning

Ku Sungula hi ku Olova: Vuyelo bya Maqhinga ya Timodelo ta Masungulo eka Dyondzo ya Muchini

Почати з простого: стратегічна перевага базових моделей у машинному навчанні

شروع کرنا آسان: مشین لرننگ میں بیس لائن ماڈلز کا اسٹریٹجک فائدہ

Oddiy boshlash: Mashinani o'rganishda asosiy modellarning strategik ustunligi

Ukuqala Okulula: UbuNcedo obuCwangcisiweyo beMifuziselo esisiseko kwiSifundo soomatshini

Ukuqala Okulula: Inzuzo Yamasu Yamamodeli Asesisekelo Ekufundeni Ngomshini

Authors: (1) Muzhaffar Hazman, University of Galway, Ireland; (2) Susan McKeever, Technological University Dublin, Ireland; (3) Josephine Griffith, University of Galway, Ireland. Table of Links Abstract and Introduction Related Works Methodology Results Limitations and Future Works Conclusion, Acknowledgments, and References A Hyperparameters and Settings B Metric: Weighted F1-Score C Architectural Details D Performance Benchmarking E Contingency Table: Baseline vs. Text-STILT E Contingency Table: Baseline vs. Text-STILT Table 8 shows the contingency table – as one would prepare for a McNemar’s Test between two classifiers (McNemar, 1947) – between the model trained with Text-STILT on 60% Memes and Baseline trained on 100% Memes available which had the most similar performance. While the two models performed similarly in terms of Weighted F1- scores, Text-STILT correctly classified a notable number of memes that Baseline did not and vice versa. Examples of such memes are discussed in Section 4.1. Furthermore, approximately 40% of memes in the testing set were incorrectly classified by both models. This suggests that these memes convey sentiment in a way that cannot be reliably predicted by either approach. This paper is under CC 4.0 license. available on arxiv

Contingency Table: Baseline vs. Text-STILT

Authors: (1) Savvas Petridis, Google Research, New York, New York, USA; (2) Ben Wedin, Google Research, Cambridge, Massachusetts, USA; (3) James Wexler, Google Research, Cambridge, Massachusetts, USA; (4) Aaron Donsbach, Google Research, Seattle, Washington, USA; (5) Mahima Pushkarna, Google Research, Cambridge, Massachusetts, USA; (6) Nitesh Goyal, Google Research, New York, New York, USA; (7) Carrie J. Cai, Google Research, Mountain View, California, USA; (8) Michael Terry, Google Research, Cambridge, Massachusetts, USA. Table Of Links Abstract & Introduction Related Work Formative Study Constitution Maker Implementation User Study Findings Discussion Conclusion and References 6 USER STUDY To understand (1) if the principle elicitation features help users write principles to steer LLM outputs and (2) what other kinds of feedback they wanted to give, we conducted a 14-participant within-subjects user study, comparing ConstitutionMaker to an ablated version without the principle elicitation features. This ablated version still offered users the ability to rewind parts of the conversation, but participants could only see one chatbot output at a time and had to write their own principles. 6.1 Procedure The overall outline of this study is as follows: (1) Participants spent 40 minutes writing principles for two separate chatbots, one with ConstitutionMaker (20 minutes) and the other with the baseline version (20 minutes), while thinking aloud. Condition order was counterbalanced, with chatbot assignment per condition also balanced. (2) After writing principles for both chatbots, participants completed a post-study questionnaire, which compared the process of writing principles with each tool. (4) Finally, in a semi-structured interview, participants described the positives and negatives of each tool and their workflow. The total time commitment of the study was 1 hour. The two chatbots participants wrote principles for were VacationBot, an assistant that helps users plan and explore different vacation options, and FoodBot, an assistant that helps you plan your meals and figure out what to eat. These two chatbots were chosen because they support tasks that most people are experienced with, so that participants could have opinions on their outputs and write principles. For both chatbots, participants were given the name and capabilities (Figure 1A), so as to focus predominantly on principle writing. Also, we balanced which chatbot went with each condition, so half of the participants used ConstitutionMaker to write principles for VacationBot, and the other half used the baseline for VacationBot. Finally, prior to using each version, participants watched a short video showing that respective tool’s features. To situate the task, participants were asked to pretend to be a chatbot designer and that they were writing principles to dictate the chatbot’s behavior so that it performs better for users. We wanted to observe their process for writing principles and see if the tools impacted how many principles they could write, so we encouraged participants to write at least 10 principles for each chatbot, to give them a concrete goal and to motivate them to write more principles. However, we emphasized that this was only to encourage them to write principles and that they should only write a principle if they thought it would be useful to future users. 6.2 Measurements and Analysis 6.2.1 Questionnaire. elicitation features help users write principles. Our questionnaire (Table 1) probes a few aspects of principle writing including participants’ perception of (1) how well the output principles effectively guide the bot, the diversity of the output principles, how easy it was to convert their feedback into principles, the efficiency of their principle writing process, and the requisite mental demand [9] for writing principles with each tool. To compare the two conditions, we conducted paired sample Wilcoxon tests with full Bonferroni correction, since the study was within-subjects and the questionnaire data was ordinal. We wanted to understand if and how well ConstitutionMaker’s principle To shed further light on which tool helped participants write principles more, we recorded the number of principles written for each condition. Moreover, to understand which of the principles elicitation features was most helpful, we recorded how often each was used during the experimental (full ConstitutionMaker) condition. To compare the average number of principles collected across the two conditions, we conducted a paired t-test. 6.2.2 Feature Usage Metrics. 6.3 Participants We recruited 14 industry professionals at a large technology company (average age = 32, 6 female and 8 male) via an email call for participation and word of mouth. These industry professionals included UX designers, software engineers, data scientists, and UX researchers. Eligible participants were those that had at least written a few LLM prompts in the past. The interviews were conducted remotely. Participants received a $25 gift card for their time. This paper is under CC 4.0 license. available on arxiv

Comparing ConstitutionMaker to Baseline: User Study Unveils Insights into Chatbot Principle Writing

Larger GPU Instances don’t always mean faster Processing The artificial intelligence, machine learning, and generative AI application's growth have swelled the demand for high-performance GPU workloads. To fulfill these needs, cloud services have introduced a broad range of instances to fulfill diverse needs. For many, has been a quick fix to increase computational power by replacing a smaller instance with a more powerful one. However, this methodology might be less effective or even pointless for GPU-intensive workloads. vertical scaling Understanding GPU Workloads GPUs are designed for high-speed mathematical calculations. These specialized computational capabilities make them a good choice or, for some applications, the only option for handling large dataset mathematical operations in machine learning (ML) and video editing. CPUs’ primary aim is to perform sequential task execution, whereas GPUs excel as workhorses when it comes to deep learning, 3D rendering, and scientific computations. The Basics of GPU Scaling Instances can be scaled vertically, horizontally, or through a combination of both as explained . The Vertical scaling strategy means upgrading to a larger server with more computational power, such as CPU, memory, and GPU. here The Horizontal scaling strategy involves adding more instances instead of upgrading the current one, hence increasing resources in parallel, very much like buying additional properties in the neighborhood to accommodate your growing family. Limitations of Vertical Scaling for GPUs Vertical scaling's effectiveness is evident when we need more CPU or memory resources, but it hits a roadblock with GPU workloads. This is because upgrading to a larger instance does not necessarily increase the GPU's computational power. For example, the instances from AWS both house the Nvidia Tesla T4 and T4G respectively. g5g.large and g4dn.large Despite any vertical scaling between these instance types, the GPU power remains unchanged, as they contain the same number of GPU cores as shown in the following image. Hence, vertical scaling will not address the actual bottleneck in GPU-intensive tasks. Practical Implications The implication is evident: when your application demands more GPU power, vertically scaling your instance will not deliver the required performance improvements. This inefficiency can lead to not just stagnant performance, but also increased costs, as you are paying for additional CPU and memory resources that your workload may never utilize. An Alternative to Vertical Scaling When the bottleneck is GPU computational capabilities, horizontal scaling becomes a more viable solution. By adding more instances to your cloud infrastructure, you can achieve a near-linear increase in GPU capabilities. Horizontal scaling with the right kind of servers will improve your infrastructure's resource utilization, thus offering a more cost-effective solution. Training an ML model with a vertically scaled server will add more CPU and memory but will not improve GPU power. Therefore, there will not be any improvement in training time. Shifting to horizontal scaling, with multiple GPU instances working in tandem, can significantly reduce the training time, showcasing the right approach to scaling for such tasks. Conclusion While vertical scaling offers a quick solution for prompt performance boost in certain scenarios, it falls short with the GPU instances option available. I highly recommend checking out for a deeper understanding of when to employ each strategy. horizontal and vertical scaling A nuanced grasp of these scaling methods is crucial for optimizing performance and effective cost management in various computational scenarios. For tasks that are GPU-bound, adding more GPU power through horizontal scaling or re-architecting the solution is often the more effective path. Even for horizontal scaling, opt for the least powered instance, such as or , to avoid paying for extra unused CPU power that is not required. Adopting the right scaling strategy, you can ensure that your infrastructure is not just robust but also cost-effective and performance-optimized for your specific computational needs. G5g.xlarge G4dn.xlarge More From the Author Learn About the Issues and Solution at Is AWS Deep Learning AMIs saving you time? A counterproductive approach hindering your progress! How Your AWS Deep Learning AMI is Holding You Back Read the full article at: A Cost-Effective Creating Deep Learning AMIs deployment with Spot Instances: Navigate parallel frontier! Deep Learning on AWS Graviton2 powered by Nvidia Tensor T4g A comprehensive guide is available at . Fine-Tune FFmpeg for Optimal Performance with detailed compilation guide: Optimize your multimedia backend for optimal performance! Tailor FFmpeg for your needs

Choosing Baseline GPU Instances: What Are the Advantages?

Authors: (1) Dominic Petrak, UKP Lab, Department of Computer Science, Technical University of Darmstadt, Germany; (2) Nafise Sadat Moosavi, Department of Computer Science, The University of Sheffield, United Kingdom; (3) Ye Tian, Wluper, London, United Kingdom; (4) Nikolai Rozanov, Wluper, London, United Kingdom; (5) Iryna Gurevych, UKP Lab, Department of Computer Science, Technical University of Darmstadt, Germany. Table of Links Abstract & Introduction Related Work Datasets Examined Manual Error Type Analysis and Taxonomies Automatic Filtering for Potentially Relevant Dialogs Statistical Analysis Evaluation and Experiments Discussion Conclusion, Limitation, Acknowledgments, and References A Integrated Error Taxonomy – Details B Error-Indicating Sentences And Phrases C Automatic Filtering – Implementation D Automatic Filtering – Sentence-Level Analysis E Task-Oriented Dialogs – Examples F Effectiveness Of Automatic Filtering – A Detailed Analysis G Inter-Annotator Agreement – Detailed Analysis H Annotation Guidelines I Hyperparameters and Baseline Experiments J Human-Human Dialogs – Examples I Hyperparameters and Baseline Experiments All baseline models were trained for five epochs. For the experiment using erroneous dialogs, we trained the models for ten epochs. We used a batch size of 32 and a learning rate of 5e − 5 with no warmup steps. As optimizer, we used the implementation of AdamW 11 (Loshchilov and Hutter, 2019) in Pytorch. Except for LLAMA (Touvron et al., 2023), we fully-finetuned all models. For LLAMA, we only finetuned the LoRA (Hu et al., 2022) weights, using a rank of 8, an alpha of 16, and a dropout rate of 0.05. Hyperparameters Table 17 shows the results of our baseline experiments using word-overlapping F1-Score and BLEU (Papineni et al., 2002). Results Compared to the results of our experiment using erroneous dialogs (Table 11), both F1-Score and BLEU are higher in case of GPT-2 and LLAMA. We attribute this to the comparatively bad quality of the erroneous dialogs. The majority of these dialogs comes from the SFC (Hancock et al., 2019) dataset, where most errors are due to out-of-context system utterances (see Section 6.1) and were ignored by users (see Section 6.2). This paper is under CC BY-NC-SA 4.0 DEED license. available on arxiv

Hyperparameters and Baseline Experiments in Dialog Systems

When the topic of InfoSec is discussed broadly, not at the level that InfoSec or cybersecurity professionals discuss it, but instead in the boardroom or at conference room tables, the same question is brought up (without fail). What can "we" do to be more secure? Invariably, vendor names & buzzwords are deployed like commandos in a 1980's Chuck Norris film. After that barrage, the next logical step is to discuss trusted third parties that can be brought in like mercenaries to "whip us into shape". This comes at a steep price, however, and I don't just mean monetarily. Often the organizations hiring these 'mercenaries' (no offense to the mercs reading this) have an advocate internally that has suggested meaningful changes for some time, but because they themselves are not viewed as a "consultant", their opinion may be invalidated. This person brings common sense ideas that are tailored to your organization, but they are not well received. You may know this person. You may BE this person! You say things like "we need a CMDB!" or "I need your buy-in to establish a routine audit of our Active Directory". You may have even suggested establishing a Golden Image for the OS's that you deploy; Windows Server, RHEL, Debian, Windows Desktop, etc... if you know this person or this person IS YOU then these words on this page are for you! First, let’s get a few housekeeping things out of the way.  I want to level set and define some of these words so we can move forward quickly. Below are a few definitions. I will explore their relationships in the following sections.... The Terms Baseline: You will find a few variations on this keyword, but the one thing to remember is this is a snapshot in time of a system’s current configuration (OS-level settings, installed apps, app configurations, users, etc...). For example, if you were to baseline a web server you might say the baseline contains the following parameters: : RedHat 8 OS Apache; MySQL, Java Applications: : (Root, Alice, Tom4) Users : Disallow Root remote login OS Settings Those parameters, and most likely many other parameters like that, would create a baseline for a system. It’s nothing more (or less) than how a system is configured at a moment in time. This baseline can be used to "look back" and see what has changed. Benchmark A benchmark is an individual check that is performed. What is a check?  It literally checks a system setting, such as if an application is installed. Is that application at XYZ patch level? Is it installed in a specific directory? All those questions are essentially a "check" and each one compromises a benchmark. The "benchmark" is a specific configuration or system setting that your organization (or industry) has determined to be a best practice. Benchmarks are a way of evaluating whether a system is adhering to known good configurations (which in turn should reduce the attack surface of that system). What do you call it when you group together a bunch of benchmarks, harden a system, and configure it just the way you need it...? Golden Image A Golden Image is typically an entire server (or desktop) system that has all the applications needed for its purpose installed, all the system-level settings configured, and all the unique changes needed to integrate it into your environment. Then you baseline this system (capture its configuration in a moment-in-time snapshot), and this baseline is now your "Golden Image". Think of this golden image as your 'gold standard, this is exactly how you would want every server deployed moving forward. It has been hardened, it is proven to be functional, and it can be managed. The Process Baseline - So how do we use these new (shiny) terms to harden and secure our environment?  First things first, we need a way to capture the existing baseline of all deployed systems. This can be done manually if you’re in a smaller environment since most servers will be close to their "out of the box" configuration if they are newer and similar. Creating a spreadsheet with hostnames on the left and then a separate doc with even a handful of things to check (your benchmarks) is a great way to get started. Remember, security is not an all-or-nothing exercise. You should think of it more along the lines of, "Am I better this week than I was last week?". Ideally, you would have an automated method to perform this query and capture all the data you need, but if you don't, there are a few ways to automate this task (ala PowerShell or Bash). I won't cover those methods here though, as they fall outside the scope of this (introductory) post.... Benchmark - Once you have a baseline captured, it’s time to look at the configuration using the benchmarks either your organization has defined or benchmarks like the ones available from the Center for Internet Security (CIS): The CIS Benchmarks main directory: HTTPS The CIS Benchmarks Windows Server: https://www.cisecurity.org/benchmark/microsoft_windows_server/ This is a crucial step. All the steps are vital, but this one is the most important. Using these benchmarks is where you discover system and configuration vulnerabilities that could lead to security incidents. Golden Image- Now that you have gathered the system configurations (Baseline) and performed the configuration checks (Benchmark) you should be ready to create a Golden Image. This image is the product of this process, this will be what is deployed into the environment and known to be secure. Practically speaking, it’s great if you can use your Golden Image as the base for all future servers (literally cloning it using your hypervisor). I would make this your #1 choice if it is at all possible. If this is not practical, then you will need to create a process and document every step completely and build in an audit step to ensure all future systems getting deployed adheres to your Golden Image. The Walkthrough I thought it would be useful to see an example of how this can work in a small (to midsize) environment, which is typically where you see the most need for this type of configuration management. 1. Gather System Baselines Let’s say this needs to be done manually, the first thing we want to do is prioritize the systems that we audit. What is the most critical system? Is it your AD Domain Controllers (DC)? Is it a line of business app server? Whatever it is, it goes first, then you pick a #2, #3, #4, etc... we want to pour our energy into where the most value can be found first! For the purpose of this walkthrough let’s say we are starting with an AD DC, or your identity management server in a typical Windows environment. For brevity, I’ll say we only want to check the following 3 parameters (benchmarks): CIS - Microsoft Windows Server 2012 R2 - 2.3.2.2: Ensure 'Audit: Shut down system immediately if unable to log security audits' is set to 'Disabled' https://workbench.cisecurity.org/benchmarks/288 r:HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Lsa -> CrashOnAuditFail -> 1; r:HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Lsa -> CrashOnAuditFail -> 2; CIS - Microsoft Windows Server 2012 R2 - 2.3.5.2: Ensure 'Domain controller: LDAP server signing requirements' is set to 'Require signing' [https://workbench.cisecurity.org/benchmarks/288] r:HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\NTDS\Parameters -> LDAPServerIntegrity -> !2; CIS - Microsoft Windows Server 2012 R2 - 2.3.6.2: Ensure 'Domain member: Digitally encrypt secure channel data (when possible)' is set to 'Enabled'https://workbench.cisecurity.org/benchmarks/288 r:HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Netlogon\Parameters -> SealSecureChannel -> 0; So, we are clear, the actual contents of these benchmarks are pointless for this walkthrough, I just need something to refer to ;) If I am doing this manually, I would either connect via remote registry, PowerShell (or RDP if no other option), and verify these registry keys exist. At this stage, all we are doing is 'looking' but not 'touching'. This is the baseline stage, no need to make changes yet. We will compare this system’s configuration to the benchmarks in the next step. 2. Identifying System Weakness via Benchmarks Once we have connected to our most important servers and gathered the information related to the (3) CIS Benchmarks above, we can now compare the system baseline to the benchmarks provided by the CIS! It is during this stage that misconfigurations or sub-par configurations will be identified, and changes will need to be implemented. Remember, good changes are thought out (tested!) and a complete back-out strategy is known prior to beginning the change. To keep track of the compliance (or non-compliance) to the benchmarks, you can use a simple spreadsheet with the hostnames on the left side (as rows) and the Benchmark names on top (as columns). Mark compliance or non-compliance how you like, just make sure it makes sense to you in a month when you come back to this (because you will get sidetracked, re-tasked, and have time away from this effort!). 3. Establish a Golden Image Once the high-priority systems, those that are most critical, have had an initial baseline and hardening completed, now would be a good time to go and create a golden image for all future server deployments. Typically, this looks like creating a new VM (let us continue to use Windows Server); then using the CIS Benchmarks to go through and modify the system to adhere to these Benchmarks. Once the benchmarks have been implemented, a complete test of the system for its intended purpose is required. This is not a small-scale test, but instead, full QA for all its intended purpose! In Closing... Deploying secure servers is easily the #1 "low-hanging" fruit for most (SMB) organizations, this effort should have appropriate levels of buy-in to ensure completion. Again, this process would ideally be mostly (or completely) automated requiring only small amounts of human interaction. I don't have any platforms in mind to recommend, but systems that gather configuration and application settings and centralize that data to streamline this process would be my first choice!

A Guide to Understanding Benchmarks, Baselines, and Golden Images

Stories, News & Mentions about BASELIME

Baselime’s Stories on HackerNoon