Modulize

modulize

Attacking Toxic Comments Kaggle Competition Using Fast.ai K is a good place to learn and practice your Machine Learning skills. It’s also a great place to find the proper dataset for your learning projects. I need a good classification NLP dataset to practice my recently learned fast.ai lesson, and I came across the . The competition is held two years ago and has long concluded, but it doesn’t hurt to submit my scores and see how well I did. This is one of the things Kaggle is great for since in the real world, it will usually be much harder to know how good or bad your model is, whereas, in Kaggle, you’ll see clearly where your performance is in the Leaderboard. aggle Toxic Comment Classification Challenge The Data Set This competition is held by The team, a research initiative founded by and Google (both a part of Alphabet). Its goal is to find out the best model that can classify multiple toxicity types in comments. The toxicity types are: Conversation AI Jigsaw toxic severe_toxic obscene threat insult indentity_hate Comments are given in a training file and a testing file . And you’ll need to predict a probability of each type of toxicity for each comment in . It is a multi-label NLP classification problem. train.cvs test.csv test.csv Look at the Data Let’s first take a look at the data. We need to import the necessary modules and do some logistics to set up the paths for our files. numpy np pandas pd fastai.text * fastai * import as # linear algebra import as # data processing, CSV file I/O (e.g. pd.read_csv) from import from import Notice here we imported everything from fastai.text and fastai modules. Are we against the software engineering best practice here? Actually, not quite. It’s rather a deliberate move in a more iterative and interactive data science kind of way. With all the library available, I can easily test and try different functions/modules without having to go back and import them every time. It will make the explore/experiment flow much more smoothly. But I digressed, let’s load the data and look at it: path = Path( )
path.ls() !mkdir data
!cp -a {path}/*.* ./data/
!ls data path = Path( )
path.ls() df = pd.read_csv(path/ )
df.head() # Kaggle store dataset in the /kaggle/input/ folder, '/kaggle/input/jigsaw-toxic-comment-classification-challenge/' # the /kaggle/input/ folder is read-only, copy away so I can also write to the folder. # make sure everything is correctly copied over '/kaggle/working/data/' # read in the data and have a peak 'train.csv' (The toxicity types are one-hot encoded) The comments are in column and all toxicity types are ‘one-hot’ encoded, we’ll have to do something about it to make it fit into our model later. comment_text (Have a look at one comment) Transfer Learning: Fine-Tune Our Language Model We’ll use transfer learning for this task, to do that, we’ll use a pre-trained model based on Wikipedia called . It is a model that’s already trained from the Wikipedia dataset(or ‘corpus’ in NLP terms) to predict the next words from a giving unfinished sentence. We’ll leverage the ‘language knowledge’ the model already learned from the Wikipedia dataset and build on top of that. To achieve the best results, we’ll need to ‘fine-tune’ the model to make it learn a bit from our ‘comments’ dataset since what people say in the comments are not necessarily the same with the more formal Wiki. Once the language model is fine-tuned, we can then use it to further do our classification task. wikitext-103 Now let’s load the training data into the fast.ai so we can start training the language model first. databunch bs = data_lm = (TextList.from_df(df, path, cols= )
                .split_by_rand_pct( )
                .label_for_lm()
                .databunch(bs=bs)) 64 # set batch size to 64, works for Kaggle Kernels 'comment_text' 0.1 We use fast.ai’s for this task. It is a very flexible and powerful way to address the challenging task of building a pipeline: loading your data into the model. It isolates the entire process into different parts/steps, each step with multiple methods/functions to adapt to different types of data and the ways data is stored. This concept is a lot like the Linux philosophy, highly modulized and with each module only do one thing but really really well. You are free to explore the wonderful API , for the above code though, it does the following things: Data Block API here 1. Import data from Pandas DataFrame named , tell the model to use as input ( ) df comment_text TextList.from_df(df, path, cols=’comment_text’) Note here I can also include the test.csv into the language model. It’s not considered ‘cheating’ since we are not using the labels, just do language model training. 2. Split the training dataset into train/validation set by random 10/90 percent. ( ) .split_by_rand_pct(0.1) 3. Ignore the given labels( since we are only fine-tuning the language model, not training the classifier yet) and use the language model’s ‘predict next word’ as labels. ( ) .label_for_lm() 4. Build the data into a , with batch size . ( ) databunch bs .databunch(bs=bs) Now let’s look at the we just built: databunch (Notice we lost all the toxicity types) Notice that the doesn’t have all the toxicity type labels since we are only fine-tuning the language model. databunch OK, time for some typical fast.ai learning rate adjustments and training: We put our into a , tell it the language model base we want to use ( ) and assign a default dropout rate of . From the graph, find the biggest downward slope and pick the middle point as our learning rate. (For a more detailed explanation of how this ‘fit_one_cycle’ magic is done, please refer to this . It is a SOTA technique of fast.ai that combines learning rate and momentum annealing). Now we can ‘unfreeze’ the model and train the entire model couple of epochs: databunch language_model_learner AWD_LSTM 0.3 LR Finder article We can look at one example of how well the model did: The result is hardly optimal. But we at least get a sentence that actually makes sense and 0.38 accuracy for predicting the next word is not bad. Ideally, we need to train a bit more epochs but for this Kaggle Kernel, I was running out of GPU quota so I stopped at 4. The result definitely has room to improve and you can try it yourself. Anyway, what we want from the language model is the encoder part, so we save it. Training the language model does take quite some time, but the good news is, for your own domain corpus, you only have to train once and later you can use it as a base for any other classification tasks. learn.save_encoder( ) # save the encoder for next step use 'fine_tuned_enc' Transfer Learning: Training the Classifier Let’s read in the test dataset: test = pd.read_csv(path/ )
test_datalist = TextList.from_df(test, cols= ) "test.csv" 'comment_text' Again, build our : databunch data_cls = (TextList.from_csv(path, , cols= , vocab=data_lm.vocab)
                .split_by_rand_pct(valid_pct= )
                .label_from_df(cols=[ , , , , , ], label_cls=MultiCategoryList, one_hot= )
                .add_test(test_datalist)
                .databunch())
data_cls.save( ) 'train.csv' 'comment_text' 0.1 'toxic' 'severe_toxic' 'obscene' 'threat' 'insult' 'identity_hate' True 'data_clas.pkl' Please note the difference this time: 1. When building the TextList , we specified vocab=data_lm.vocab , this way we make sure we are using the same vocabulary and our training on the language model can be properly applied to the classifier model. 2. We now use all our toxicity styles labels ( ) .label_from_df(cols=[‘toxic’, ‘severe_toxic’,’obscene’, ‘threat’, ‘insult’, ‘identity_hate’],label_cls=MultiCategoryList, one_hot=True), 3. We added our test set here. ( ) .add_test(test_datalist) Now look at our classifier : databunch (Note that now we have all the toxicity styles labels) We’ll put the into the model and load the encoder we learned from the language model. Finally, time to put everything together! databunch text_classifier_learner learn = text_classifier_learner(data_clas, AWD_LSTM, drop_mult= )
learn.load_encoder( ) 0.5 'fine_tuned_enc' Again, find the best learning rate and train one cycle: Train a bit more cycles and unfreeze: See the results: Off by one, but overall the prediction is OK. For the purpose of reference, I submitted the prediction to Kaggle and get a 0.98098 Public Score (land in the middle of the Public Leader Board). The result is not optimal but like I said I didn’t train all the way due to limited GPU. The purpose of this article is to show you the whole process of using fast.ai to tackle multi-labels text classification problem. The real challenge here is to load the data into the model using Data Block API. Conclusion I hope you learned a thing or two from this article. Fast.ai is really a lean, flexible and powerful library. For the things it can do (like image/text classification, tabular data, collaborative filtering, etc.), it does it very well. It is not as extensive as Keras, but it’s very sharp and focused. Kind of like Vim and Emacs if you are familiar with the command line text editor war. 😜 You can find the Kaggle Kernel . here Any feedback or constructive criticism is welcomed. You can either find me on Twitter or my blog site . @lymenlee wayofnumbers.com

How to Build a Multi-label NLP Classifier from Scratch

Author:
(1) CALLA TSCHANZ. Table of Links Abstract and Intro
Background on tropical perspective
The expanded construction
GIT stability
Stack perspective
The canonical moduli stack
References 6. The canonical moduli stack 6.1 Properness and Deligne-Mumford property Existence and uniqueness of limits for special objects. We need to establish some definitions before we prove the following auxiliary result on existence and uniqueness of limits for special elements, i.e. when the fibre Xη over the generic point of S is a modified special fibre itself. We start by proving existence and uniqueness of limits in the first case using the valuative criterion. Let V denote the irreducible component of Xη in the interior of which P lies. Notice that since P tends towards a codimension greater or equal to one stratum of X, then in order for its limit to be smoothly supported in an extension of (Zη, Xη), it will be necessary to expand out at least one ∆-component in this extension. There exists a smoothing from the interior of V in the fibre over the generic point to the interior of this expanded ∆-component in such an extension of (Zη, Xη) if and only if this ∆-component is equal to V in the fibre over the generic point. Moreover, if there is no such ∆-component equal to V, then none of the x, y or z coordinates can tend towards zero (because both sides of the defining equations must tend towards zero). Deligne-Mumford property. Finally we show that both stacks of stable objects constructed have finite automorphisms. Proof. This follows directly from the results of this section. 6.2 An isomorphism of stacks We will need also the following result from Alper and Kresch [AK16]. Now we are in a position to prove the following theorem: This paper is available on arxiv under CC 4.0 license. Author: (1) CALLA TSCHANZ. Author: Author: (1) CALLA TSCHANZ. Table of Links Abstract and Intro Background on tropical perspective The expanded construction GIT stability Stack perspective The canonical moduli stack References Abstract and Intro Abstract and Intro Background on tropical perspective Background on tropical perspective The expanded construction The expanded construction GIT stability GIT stability Stack perspective Stack perspective The canonical moduli stack The canonical moduli stack References References 6. The canonical moduli stack 6.1 Properness and Deligne-Mumford property Existence and uniqueness of limits for special objects. We need to establish some definitions before we prove the following auxiliary result on existence and uniqueness of limits for special elements, i.e. when the fibre Xη over the generic point of S is a modified special fibre itself. Existence and uniqueness of limits for special objects. We start by proving existence and uniqueness of limits in the first case using the valuative criterion. Let V denote the irreducible component of Xη in the interior of which P lies. Notice that since P tends towards a codimension greater or equal to one stratum of X, then in order for its limit to be smoothly supported in an extension of (Zη, Xη), it will be necessary to expand out at least one ∆-component in this extension. There exists a smoothing from the interior of V in the fibre over the generic point to the interior of this expanded ∆-component in such an extension of (Zη, Xη) if and only if this ∆-component is equal to V in the fibre over the generic point. Moreover, if there is no such ∆-component equal to V, then none of the x, y or z coordinates can tend towards zero (because both sides of the defining equations must tend towards zero). Deligne-Mumford property. Finally we show that both stacks of stable objects constructed have finite automorphisms. Deligne-Mumford property. Proof . This follows directly from the results of this section. Proof 6.2 An isomorphism of stacks We will need also the following result from Alper and Kresch [AK16]. Now we are in a position to prove the following theorem: This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Expansions for Hilbert Schemes: the Canonical Moduli Stack

হিলবার্ট স্কিমগুলির সম্প্রসারণ: ক্যানোনিকাল মডুলি স্ট্যাক

Erweiterungen für Hilbert-Schemata: der kanonische Modulstapel

Ampliaciones para esquemas de Hilbert: la pila de módulos canónicos

Extensions pour les schémas de Hilbert : la pile de modules canoniques

हिल्बर्ट योजनाओं के लिए विस्तार: कैनोनिकल मोडुली स्टैक

ヒルベルトスキームの展開: 標準モジュライスタック

힐베르트 방식의 확장: 정규 모듈러스 스택(Canonical Moduli Stack)

Expansões para esquemas de Hilbert: a pilha de módulos canônicos

Разложения схем Гильберта: стек канонических модулей

Hilbert Şemaları için Genişletmeler: Kanonik Modül Yığını

Mở rộng cho các sơ đồ Hilbert: Ngăn xếp mô-đun Canonical

希尔伯特方案的扩展：规范模数堆栈

Authors:
(1) Wahei Hara;
(2) Yuki Hirano. Table of Links Abstract and Intro
Exchanges and Mutations of modifying modules
Quasi-symmetric representation and GIT quotient
Main results
Applications to Calabi-Yau complete intersections
Appendix A. Matrix factorizations
Appendix B. List of Notation
References 2. Exchanges and Mutations of modifying modules 2.1. Noncommutative crepant resolution. The present section recalls the definition of some basic notions that are studied in this article. (1) A reflexive R-module M is called a modifying module if EndR(M) is a (maximal) Cohen-Macaulay R-module. (2) We say that a reflexive module M gives a noncommutative crepant resolution (=NCCR) Λ = EndR(M) if M is modifying and the algebra Λ has finite global dimension. Remark 2.4. Note that our definition of NCCR is different from the one in [Van3] or [IW1]. However, if R is d-sCY, our definition is equivalent to other definitions. See [Van3, Lemma 4.2] or [IW1, Lemma 2.23]. from K ∈ addL such that the induced morphism α ◦ (−): Hom(N, K) → Hom(N, M) is surjective. If L = N, we just call α a right (addL)-approximation of M. A right (add L)N - approximation α: K → M of M is said to be minimal if any endomorphism φ ∈ End(K) satisfying α◦φ = α is an automorphism, and we say that α is reduced if any direct summand K′ of K does not contained in Ker(α). Note that if a right approximation is minimal, it is reduced, and in the case when R is complete local, the converse also holds. Definition 2.6. Let R be a normal d-sCY, and let M, N, L ∈ ref R. Lemma 2.7. Notation is same as above (1) If L ′ ∈ addL, there is an inclusion which remains to be true when restricting to reduced exchanges. (2) If N′ ∈ add N, there is an inclusion which remains to be true when restricting to reduced exchanges. (3) For another full subcategory S ′ ⊆ ref R, there is an inclusion If R is complete local, the similar inclusion also holds for reduced exchanges. Proof. (1), (2) and the first assertion in (3) are obvious. The second assertion in (3) follows from the fact that, if R is complete local, two approximations α: K → M and α ′ : K′ → M′ are reduced if and only if α ⊕ α ′ : K ⊕ K′ → M ⊕ M′ is reduced. Proof. Assume that Hom(N, M ⊕ N) is Cohen-Macaulay, and consider an exact sequence 0 → F Ker α → FK → FM → 0. Now applying the functor Hom(−, FR) to this sequence together with the reflexive equivalence proves that the dual sequence 0 → M∗ → K∗ → (Ker α) is exact. 0 → Hom(FM, FN) → Hom(FK, FN) → Hom(F Ker α, FN) → 0 remains to be exact. Since all modules in the original sequence are reflexive, the reflexive equivalence and the duality yield an isomorphism and similar isomorphisms for K and Ker α, which imply the exactness of the sequence 0 → Hom(N ∗ , M∗ ) → Hom(N ∗ , K∗ ) → Hom(N ∗ ,(Ker α) ∗ ) → 0. Thus the dual morphism K∗ → (Ker α) ∗ is a right (add L ∗ )N∗ -approximation with the kernel M∗ , which proves the first assertion. The second assertion follows from a similar argument. The following says that exchanging a direct summand of a modifying module gives a new modifying module in nice situations. Lemma 2.10. Let M ∈ ref R. The following equivalence holds. M ∈ CM R ⇐⇒ M∗ ∈ CM R Proof. We may assume that R is local. Since M is reflexive, it is enough to show the direction (⇒). Since R is Gorenstein, its injective dimension is finite. Thus the result follows from [BH, Proposition 3.3.3 (b)]. Lemma 2.11. Let R be a Gorenstein normal ring, and let M, N ∈ ref R. Then Proof. It is enough to prove the direction (⇒). Assume that Hom(M, N) ∈ CM R. Then Lemma 2.10 implies that Hom(M, N) ∗ ∈ CM R. But by Lemma [IW1, Lemma 2.9], there is an isomorphism Hom(M, N) ∗ ∼= Hom(N, M), which shows that Hom(N, M) ∈ CM R. The proof for the case when m < 0 is similar. Remark 2.13. Since a right approximation is not unique in general, neither is right/left mutation. However, right/left mutation is unique up to additive closure [IW1, Lemma 6.2], and if R is complete local, minimal mutations are unique up to isomorphism. Theorem 2.14 ([IW1, Proposition 6.5, Theorem 6.8, Theorem 6.10]). Let M ∈ ref R be a modifying R-module. 2.3. Tilting bundles and mutations. This section discusses tilting bundles over algebraic stacks. We start from recalling some basic facts on the derived categories of algebraic stacks. This paper is available on arxiv under CC0 1.0 DEED license. Authors: (1) Wahei Hara; (2) Yuki Hirano. Authors: Authors: (1) Wahei Hara; (2) Yuki Hirano. Table of Links Abstract and Intro Exchanges and Mutations of modifying modules Quasi-symmetric representation and GIT quotient Main results Applications to Calabi-Yau complete intersections Appendix A. Matrix factorizations Appendix B. List of Notation References Abstract and Intro Abstract and Intro Exchanges and Mutations of modifying modules Exchanges and Mutations of modifying modules Quasi-symmetric representation and GIT quotient Quasi-symmetric representation and GIT quotient Main results Main results Applications to Calabi-Yau complete intersections Applications to Calabi-Yau complete intersections Appendix A. Matrix factorizations Appendix A. Matrix factorizations Appendix B. List of Notation Appendix B. List of Notation References References 2. Exchanges and Mutations of modifying modules 2.1. Noncommutative crepant resolution. The present section recalls the definition of some basic notions that are studied in this article. Noncommutative crepant resolution. (1) A reflexive R-module M is called a modifying module if EndR(M) is a (maximal) Cohen-Macaulay R-module. (2) We say that a reflexive module M gives a noncommutative crepant resolution (=NCCR) Λ = EndR(M) if M is modifying and the algebra Λ has finite global dimension. Remark 2.4. Note that our definition of NCCR is different from the one in [Van3] or [IW1]. However, if R is d-sCY, our definition is equivalent to other definitions. See [Van3, Lemma 4.2] or [IW1, Lemma 2.23]. Remark 2.4. from K ∈ addL such that the induced morphism α ◦ (−): Hom(N, K) → Hom(N, M) is surjective. If L = N, we just call α a right (addL)-approximation of M. A right (add L)N - approximation α: K → M of M is said to be minimal if any endomorphism φ ∈ End(K) satisfying α◦φ = α is an automorphism, and we say that α is reduced if any direct summand K′ of K does not contained in Ker(α). Note that if a right approximation is minimal, it is reduced, and in the case when R is complete local, the converse also holds. Definition 2.6. Let R be a normal d-sCY, and let M, N, L ∈ ref R. Definition 2.6. Lemma 2.7. Notation is same as above Lemma 2.7. Notation is same as above (1) If L ′ ∈ addL, there is an inclusion If L ′ ∈ addL, there is an inclusion which remains to be true when restricting to reduced exchanges. (2) If N′ ∈ add N, there is an inclusion If N′ ∈ add N, there is an inclusion which remains to be true when restricting to reduced exchanges. which remains to be true when restricting to reduced exchanges. (3) For another full subcategory S ′ ⊆ ref R, there is an inclusion For another full subcategory S ′ ⊆ ref R, there is an inclusion If R is complete local, the similar inclusion also holds for reduced exchanges. If R is complete local, the similar inclusion also holds for reduced exchanges. Proof . (1), (2) and the first assertion in (3) are obvious. The second assertion in (3) follows from the fact that, if R is complete local, two approximations α: K → M and α ′ : K′ → M′ are reduced if and only if α ⊕ α ′ : K ⊕ K′ → M ⊕ M′ is reduced. Proof Proof . Assume that Hom(N, M ⊕ N) is Cohen-Macaulay, and consider an exact sequence Proof 0 → F Ker α → FK → FM → 0. Now applying the functor Hom(−, FR) to this sequence together with the reflexive equivalence proves that the dual sequence 0 → M∗ → K∗ → (Ker α) is exact. 0 → Hom(FM, FN) → Hom(FK, FN) → Hom(F Ker α, FN) → 0 remains to be exact. Since all modules in the original sequence are reflexive, the reflexive equivalence and the duality yield an isomorphism and similar isomorphisms for K and Ker α, which imply the exactness of the sequence 0 → Hom(N ∗ , M∗ ) → Hom(N ∗ , K∗ ) → Hom(N ∗ ,(Ker α) ∗ ) → 0. Thus the dual morphism K∗ → (Ker α) ∗ is a right (add L ∗ )N∗ -approximation with the kernel M∗ , which proves the first assertion. The second assertion follows from a similar argument. The following says that exchanging a direct summand of a modifying module gives a new modifying module in nice situations. Lemma 2.10. Let M ∈ ref R. The following equivalence holds. Lemma 2.10. Let M ∈ ref R. The following equivalence holds. M ∈ CM R ⇐⇒ M∗ ∈ CM R Proof. We may assume that R is local. Since M is reflexive, it is enough to show the direction (⇒). Since R is Gorenstein, its injective dimension is finite. Thus the result follows from [BH, Proposition 3.3.3 (b)]. Lemma 2.11. Let R be a Gorenstein normal ring, and let M, N ∈ ref R. Then Lemma 2.11. Proof . It is enough to prove the direction (⇒). Assume that Hom(M, N) ∈ CM R. Then Lemma 2.10 implies that Hom(M, N) ∗ ∈ CM R. But by Lemma [IW1, Lemma 2.9], there is an isomorphism Hom(M, N) ∗ ∼= Hom(N, M), which shows that Hom(N, M) ∈ CM R. Proof The proof for the case when m < 0 is similar. Remark 2.13. Since a right approximation is not unique in general, neither is right/left mutation. However, right/left mutation is unique up to additive closure [IW1, Lemma 6.2], and if R is complete local, minimal mutations are unique up to isomorphism. Theorem 2.14 ([IW1, Proposition 6.5, Theorem 6.8, Theorem 6.10]). Let M ∈ ref R be a modifying R-module. 2.3. Tilting bundles and mutations. This section discusses tilting bundles over algebraic stacks. We start from recalling some basic facts on the derived categories of algebraic stacks. This paper is available on arxiv under CC0 1.0 DEED license. This paper is available on arxiv under CC0 1.0 DEED license. available on arxiv

Mutations of noncommutative crepant resolutions: Exchanges & Mutations of modifying modules

নন-কমিউটেটিভ ক্রেপ্যান্ট রেজোলিউশনের মিউটেশন: পরিবর্তনকারী মডিউলগুলির বিনিময় এবং মিউটেশন

Mutationen nichtkommutativer krepanter Auflösungen: Austausch und Mutationen modifizierender Module

Mutaciones de resoluciones crepantes no conmutativas: intercambios y mutaciones de módulos modificadores

Mutations de résolutions crépantes non commutatives : Echanges & Mutations de modules modificateurs

गैर-विनिमेय क्रेपेंट संकल्पों के उत्परिवर्तन: संशोधित मॉड्यूल के आदान-प्रदान और उत्परिवर्तन

非可換クレパント解決の突然変異: 修正モジュールの交換と突然変異

비가환적 크레판트 해결의 돌연변이: 수정 모듈의 교환 및 돌연변이

Mutações de resoluções crepantes não comutativas: Trocas e mutações de módulos modificadores

Мутации некоммутативных крепантных резолюций: обмены и мутации модифицирующих модулей

Değişmeyen crepant çözünürlüklerin mutasyonları: Değiştirici modüllerin Değişimleri ve Mutasyonları

Đột biến của độ phân giải crepant không giao hoán: Trao đổi & Đột biến của các mô-đun sửa đổi

非交换性 crepant 解析的突变：修改模块的交换和突变

Authors:
(1) Xiao-Yang Liu, Hongyang Yang, Columbia University (xl2427,hy2500@columbia.edu);
(2) Jiechao Gao, University of Virginia (jg5ycn@virginia.edu);
(3) Christina Dan Wang (Corresponding Author), New York University Shanghai (christina.wang@nyu.edu). Table of Links Abstract and 1 Introduction 2 Related Works and 2.1 Deep Reinforcement Learning Algorithms 2.2 Deep Reinforcement Learning Libraries and 2.3 Deep Reinforcement Learning in Finance 3 The Proposed FinRL Framework and 3.1 Overview of FinRL Framework 3.2 Application Layer 3.3 Agent Layer 3.4 Environment Layer 3.5 Training-Testing-Trading Pipeline 4 Hands-on Tutorials and Benchmark Performance and 4.1 Backtesting Module 4.2 Baseline Strategies and Trading Metrics 4.3 Hands-on Tutorials 4.4 Use Case I: Stock Trading 4.5 Use Case II: Portfolio Allocation and 4.6 Use Case III: Cryptocurrencies Trading 5 Ecosystem of FinRL and Conclusions, and References 4 HANDS-ON TUTORIALS AND BENCHMARK PERFORMANCE We provide hands-on tutorials and reproduce existing works as use cases. Their configurations and commands are available on Github. 4.1 Backtesting Module Backtesting plays a key role in evaluating a trading strategy. FinRL library provides an automated backtesting module based on Quantopian pyfolio package [36]. It is easy to use and consists of various individual plots that provide a comprehensive image of the performance. In order to facilitate users, FinRL also incorporates market frictions, market liquidity and the investor’s degree of risk-aversion. 4.1.1 Incorporating Trading Constraints. Transaction costs incur when executing a trade, such as broker commissions and the SEC fee. We allow users to treat transaction costs as parameters in the environments: 1). Flat fee is a fixed amount per trade; and 2). Per share percentage is a percentage rate for every share, e.g., 0.1% or 0.2% are most commonly used. Moreover, we need to consider market liquidity for stock trading, e.g., the bid-ask spread that is the difference between the best bid and ask prices. In our environment, users can add the bid-ask spread as a parameter. For different levels of risk-aversion, users can add the standard deviation of the portfolio returns into the reward function or use a risk-adjusted Sharpe ratio as the reward function. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Xiao-Yang Liu, Hongyang Yang, Columbia University (xl2427,hy2500@columbia.edu); (2) Jiechao Gao, University of Virginia (jg5ycn@virginia.edu); (3) Christina Dan Wang (Corresponding Author), New York University Shanghai (christina.wang@nyu.edu). Authors: Authors: (1) Xiao-Yang Liu, Hongyang Yang, Columbia University (xl2427,hy2500@columbia.edu); (2) Jiechao Gao, University of Virginia (jg5ycn@virginia.edu); (3) Christina Dan Wang ( Corresponding Author ), New York University Shanghai (christina.wang@nyu.edu). Corresponding Author Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Related Works and 2.1 Deep Reinforcement Learning Algorithms 2 Related Works and 2.1 Deep Reinforcement Learning Algorithms 2.2 Deep Reinforcement Learning Libraries and 2.3 Deep Reinforcement Learning in Finance 2.2 Deep Reinforcement Learning Libraries and 2.3 Deep Reinforcement Learning in Finance 3 The Proposed FinRL Framework and 3.1 Overview of FinRL Framework 3 The Proposed FinRL Framework and 3.1 Overview of FinRL Framework 3.2 Application Layer 3.2 Application Layer 3.3 Agent Layer 3.3 Agent Layer 3.4 Environment Layer 3.4 Environment Layer 3.5 Training-Testing-Trading Pipeline 3.5 Training-Testing-Trading Pipeline 4 Hands-on Tutorials and Benchmark Performance and 4.1 Backtesting Module 4 Hands-on Tutorials and Benchmark Performance and 4.1 Backtesting Module 4.2 Baseline Strategies and Trading Metrics 4.2 Baseline Strategies and Trading Metrics 4.3 Hands-on Tutorials 4.3 Hands-on Tutorials 4.4 Use Case I: Stock Trading 4.4 Use Case I: Stock Trading 4.5 Use Case II: Portfolio Allocation and 4.6 Use Case III: Cryptocurrencies Trading 4.5 Use Case II: Portfolio Allocation and 4.6 Use Case III: Cryptocurrencies Trading 5 Ecosystem of FinRL and Conclusions, and References 5 Ecosystem of FinRL and Conclusions, and References 4 HANDS-ON TUTORIALS AND BENCHMARK PERFORMANCE We provide hands-on tutorials and reproduce existing works as use cases. Their configurations and commands are available on Github. 4.1 Backtesting Module Backtesting plays a key role in evaluating a trading strategy. FinRL library provides an automated backtesting module based on Quantopian pyfolio package [36]. It is easy to use and consists of various individual plots that provide a comprehensive image of the performance. In order to facilitate users, FinRL also incorporates market frictions, market liquidity and the investor’s degree of risk-aversion. 4.1.1 Incorporating Trading Constraints. Transaction costs incur when executing a trade, such as broker commissions and the SEC fee. We allow users to treat transaction costs as parameters in the environments: 1). Flat fee is a fixed amount per trade; and 2). Per share percentage is a percentage rate for every share, e.g., 0.1% or 0.2% are most commonly used. 4.1.1 Incorporating Trading Constraints. Flat fee Per share percentage Moreover, we need to consider market liquidity for stock trading, e.g., the bid-ask spread that is the difference between the best bid and ask prices. In our environment, users can add the bid-ask spread as a parameter. For different levels of risk-aversion, users can add the standard deviation of the portfolio returns into the reward function or use a risk-adjusted Sharpe ratio as the reward function. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

How FinRL's Backtesting Module Enhances Trading Strategy Validation

Hey there, fellow Android developers! 👋 Are you tired of the hassle that comes with managing assisted injection in your modular app? Well, buckle up because I've got some exciting news for you! 🚀 Introducing Anvil Utils - a game-changing library that will make your life easier when working with Dagger and Anvil in multi-module projects. The Struggle Is Real Picture this: you have a beautifully architected modular app with api and implementation modules per feature. Everything is going smoothly until you encounter a situation where your feature contains a factory for an entity and want to expose it in an api, and you need to make this factory available within the Dagger graph. 😰 Let's walk through the ways to do that. Bare Dagger Approach For instance, you have such public API for your feature module: interface MyClass {
	interface Factory {
	    fun create(param1: String, param2: Int): MyClass
	}
} Traditionally, you'd have to manually create an @AssistedFactory in the implementation module, inherit from the public interface in the API module, and write a Dagger module which binds assisted factory to it's public interface. Also, you would need to maintain both factory interfaces - public one and implementation one. Here's an example: class DefaultMyClass @AssistedInject constructor(
    @Assisted param1: String,
    @Assisted param2: Int
) : MyClass {
	@AssistedFactory
	interface Factory: MyClass.Factory {
		override fun create(param1: String, param2: Int): DefaultMyClass
	}
}

@Module
interface MyClassFactoryBindingModule {
	@Binds
	fun bindFactory(impl: DefaultMyClass.Factory): MyClass.Factory
} Seems like a lot of work, doesn't it? Let's make it easier to maintain step by step Introducing Anvil Anvil is a Kotlin compiler plugin which helps to drastically reduce the boilerplate needed to use Dagger 2 in your application. Also, if properly configured, it can improve build times in your application by removing the requirement to run Dagger 2 annotation processor in your feature modules. I will not dive deep into what Anvil is and why you should consider using it. That's a topic for another day. Please read through the Anvil readme here to get what it is about. https://github.com/square/anvil Thanks to Anvil, you can get rid of the binding module using @ContributesBinding: class DefaultMyClass @AssistedInject constructor(
    @Assisted param1: String,
    @Assisted param2: Int
) : MyClass {
	@ContributesBinding(AppGraph::class, MyClass.Factory::class)
	@AssistedFactory
	interface Factory: MyClass.Factory {
		override fun create(param1: String, param2: Int): DefaultMyClass
	}
} Anvil will generate corresponding module which will bind DefaultMyClass.Factory to MyClass.Factory for you. This lowers the amount of boilerplate, but it's still there! You have to manually keep public factory interface and implementation interface aligned with the constructor! Can we go further and reduce the amount of work needed to support this use case? Sure! @ContributesAssistedFactory to the Rescue! Fear not, my friend! With Anvil Utils, you can say goodbye to the manual labor of creating assisted factories. The @ContributesAssistedFactory annotation is here to save the day! 🦸‍♂️ This powerful annotation automatically generates assisted factories for your annotated classes and contributes them to the specified scope as bindings of the provided factory type. It's like having a personal assistant that takes care of all the heavy lifting for you! You now can remove the @AssistedFactory interface completely and annotate the DefaultMyClass with @ContributesAssistedFactory: @ContributesAssistedFactory(AppGraph::class, MyClass.Factory::class)
class DefaultMyClass @AssistedInject constructor(
    @Assisted param1: String,
    @Assisted param2: Int
) : MyClass And voila! Anvil Utils will generate the necessary assisted factory for you, without breaking a sweat. 😎 In the Wild Example To demonstrate the power of this library, I've made a huge pull request to beautiful open-source project - companion app for the Flipper - Tamagochi for hackers. Please, take a look if you're interested: https://github.com/flipperdevices/Flipper-Android-App/pull/79 Embrace the Power of Anvil Utils So, what are you waiting for? Give Anvil Utils a try in your next modular Android project and experience the joy of hassle-free assisted injection. Your codebase will thank you, and your fellow developers will be in awe of your newfound superpowers! 💪 Happy coding, and may the power of Anvil Utils be with you! 🚀 P.S. Don't hesitate to open feature requests - I'm eager to improve the library and add more features. Hey there, fellow Android developers! 👋 Are you tired of the hassle that comes with managing assisted injection in your modular app? Well, buckle up because I've got some exciting news for you! 🚀 Introducing Anvil Utils - a game-changing library that will make your life easier when working with Dagger and Anvil in multi-module projects. Anvil Utils Anvil Utils Anvil The Struggle Is Real Picture this: you have a beautifully architected modular app with api and implementation modules per feature. Everything is going smoothly until you encounter a situation where your feature contains a factory for an entity and want to expose it in an api , and you need to make this factory available within the Dagger graph. 😰 api implementation api Let's walk through the ways to do that. Bare Dagger Approach For instance, you have such public API for your feature module: interface MyClass {
	interface Factory {
	    fun create(param1: String, param2: Int): MyClass
	}
} interface MyClass {
	interface Factory {
	    fun create(param1: String, param2: Int): MyClass
	}
} Traditionally, you'd have to manually create an @AssistedFactory in the implementation module, inherit from the public interface in the API module, and write a Dagger module which binds assisted factory to it's public interface. Also, you would need to maintain both factory interfaces - public one and implementation one. Here's an example: @AssistedFactory class DefaultMyClass @AssistedInject constructor(
    @Assisted param1: String,
    @Assisted param2: Int
) : MyClass {
	@AssistedFactory
	interface Factory: MyClass.Factory {
		override fun create(param1: String, param2: Int): DefaultMyClass
	}
}

@Module
interface MyClassFactoryBindingModule {
	@Binds
	fun bindFactory(impl: DefaultMyClass.Factory): MyClass.Factory
} class DefaultMyClass @AssistedInject constructor(
    @Assisted param1: String,
    @Assisted param2: Int
) : MyClass {
	@AssistedFactory
	interface Factory: MyClass.Factory {
		override fun create(param1: String, param2: Int): DefaultMyClass
	}
}

@Module
interface MyClassFactoryBindingModule {
	@Binds
	fun bindFactory(impl: DefaultMyClass.Factory): MyClass.Factory
} Seems like a lot of work, doesn't it? Let's make it easier to maintain step by step Introducing Anvil Anvil is a Kotlin compiler plugin which helps to drastically reduce the boilerplate needed to use Dagger 2 in your application. Anvil Also, if properly configured, it can improve build times in your application by removing the requirement to run Dagger 2 annotation processor in your feature modules. I will not dive deep into what Anvil is and why you should consider using it. That's a topic for another day. Please read through the Anvil readme here to get what it is about. https://github.com/square/anvil https://github.com/square/anvil Thanks to Anvil, you can get rid of the binding module using @ContributesBinding : @ContributesBinding class DefaultMyClass @AssistedInject constructor(
    @Assisted param1: String,
    @Assisted param2: Int
) : MyClass {
	@ContributesBinding(AppGraph::class, MyClass.Factory::class)
	@AssistedFactory
	interface Factory: MyClass.Factory {
		override fun create(param1: String, param2: Int): DefaultMyClass
	}
} class DefaultMyClass @AssistedInject constructor(
    @Assisted param1: String,
    @Assisted param2: Int
) : MyClass {
	@ContributesBinding(AppGraph::class, MyClass.Factory::class)
	@AssistedFactory
	interface Factory: MyClass.Factory {
		override fun create(param1: String, param2: Int): DefaultMyClass
	}
} Anvil will generate corresponding module which will bind DefaultMyClass.Factory to MyClass.Factory for you. Anvil DefaultMyClass.Factory MyClass.Factory This lowers the amount of boilerplate, but it's still there! You have to manually keep public factory interface and implementation interface aligned with the constructor! Can we go further and reduce the amount of work needed to support this use case? Sure! @ContributesAssistedFactory to the Rescue! Fear not, my friend! With Anvil Utils , you can say goodbye to the manual labor of creating assisted factories. The @ContributesAssistedFactory annotation is here to save the day! 🦸‍♂️ Anvil Utils Anvil Utils @ContributesAssistedFactory This powerful annotation automatically generates assisted factories for your annotated classes and contributes them to the specified scope as bindings of the provided factory type. It's like having a personal assistant that takes care of all the heavy lifting for you! You now can remove the @AssistedFactory interface completely and annotate the DefaultMyClass with @ContributesAssistedFactory : @AssistedFactory DefaultMyClass @ContributesAssistedFactory @ContributesAssistedFactory(AppGraph::class, MyClass.Factory::class)
class DefaultMyClass @AssistedInject constructor(
    @Assisted param1: String,
    @Assisted param2: Int
) : MyClass @ContributesAssistedFactory(AppGraph::class, MyClass.Factory::class)
class DefaultMyClass @AssistedInject constructor(
    @Assisted param1: String,
    @Assisted param2: Int
) : MyClass And voila! Anvil Utils will generate the necessary assisted factory for you, without breaking a sweat. 😎 Anvil Utils Anvil Utils In the Wild Example To demonstrate the power of this library, I've made a huge pull request to beautiful open-source project - companion app for the Flipper - Tamagochi for hackers. Please, take a look if you're interested: https://github.com/flipperdevices/Flipper-Android-App/pull/79 https://github.com/flipperdevices/Flipper-Android-App/pull/79 Embrace the Power of Anvil Utils So, what are you waiting for? Give Anvil Utils a try in your next modular Android project and experience the joy of hassle-free assisted injection. Your codebase will thank you, and your fellow developers will be in awe of your newfound superpowers! 💪 Anvil Utils Anvil Utils Happy coding, and may the power of Anvil Utils be with you! 🚀 Anvil Utils Anvil Utils P.S. Don't hesitate to open feature requests - I'm eager to improve the library and add more features. P.S.

Effortless Assisted Injection in Multi-Module Android Projects: Introducing Anvil Utils

DUBAI, UAE, April 18th, 2024/Chainwire/--SeedHunter, is a dedicated platform that facilitates and fosters collaborations between influencers and crypto projects to gain traction in the web3 community. The company has a new and fresh perspective on how to match high-potential influencers and KOLs with respective crypto projects and investors in a lucrative, scalable, and secure way. SeedHunter launched on April 18th 2024, its debut service: The Marketing Module. This is a platform that allows crypto projects to manage and maintain their influencer marketing campaigns more efficiently, securely, and effectively. The marketing campaigns launched can be uniquely customized to the crypto project’s needs in terms of content, deliverables, and more. The platform then uses an AI-based data model to evaluate, rate, and match reliable KOLs to crypto projects. SeedHunter will also assess influencer performance and define pricing directly using an advanced algorithm based on metrics of engagement, impression rates, number of followers, and the compatibility of the KOL with the campaign they manage - all of these standards will ensure that the right influencer is matched to the user's target audience needs. This ensures a fully transparent, fair, and lucrative influencer marketing position for influencers of all kinds. Crypto projects can also benefit from several built-in marketing campaign tools to drive engagement - including consultancy and account management as needed. SeedHunter takes care of all the tedious admin to provide a simplified, insightful, and efficient platform to view potential influencer and KOL partnerships. Dariusz Kowalczyk, the co-founder and CTO of SeedHunter, has stated that the platform “does the due diligence to find reliable and relevant KOLs that match your target audience’s interests. This saves time and energy while pulling the focus onto the customer.” SeedHunter also makes cryptocurrency projects a lot more accessible by simplifying processes. Influencer campaigns can be created in a matter of minutes through a centralized platform - giving the community automated contracts, KOL reviews, and delivery checks. Payments will also be based on deliveries to ensure cost-effective marketing. The web3 community can easily use SeedHunter to curate the ideal influencer campaign for their project and push growth. To celebrate its official launch, SeedHunter is diving right in with a platform campaign open to all interested KOLs. This provides an opportunity for influencers to get involved from the get-go. SeedHunter has also three additional, high-profile projects waiting to be launched. These span the exciting realms of blockchain, entertainment, and gaming, offering KOLs a chance to contribute to cutting-edge initiatives and propel their marketing efforts. The crypto industry is uniquely disruptive and as such, it needs platforms like SeedHunter that offer scalable solutions from the very beginning for a lasting chance of success. With automated, simplified, and user-friendly interfaces, SeedHunter provides a seamless experience for influencers and crypto projects alike. Co-founder and CEO, Junaid Dar has said, “The SeedHunter platform will act as a canvas for prospective reliable influencers and KOLs to connect to crypto projects that are looking to gain traction in the market in a way that will simplify, secure, and inspire the industry.” SeedHunter is built on the values of trust, efficiency, collaboration, dynamics, and prosperity. This has helped the company establish a clear, guided vision for the future of the crypto influencer market. The platform’s ultimate mission is to ensure an easier, scalable, and more secure channel through which innovative crypto projects can be matched to specific influencers and investors in a way that adds value and clarity to a dynamic cryptocurrency market. SeedHunter invites crypto project managers and influencers of all sizes to take advantage of the Marketing Module that simplifies collaborations in a mutually beneficial, fair, and secure way. SeedHunter is also open to partnerships in the cryptocurrency industry that would help to expand and establish the vision of efficient, secure, and reliable collaborations across the web3 community. Moving forward, SeedHunter will also be launching its Funding Module in June 2024 which will allow crypto projects to find the necessary seed capital needed to gain traction. In this project, SeedHunter will act as a broker between the community and crypto projects that need funding. This project ensures the involvement of smaller influencers as well and allows the project to engage with communities and participate in projects for only $50. This ensures safety, transparency, and safety for start-up crypto projects looking to change the industry. About SeedHunter SeedHunter was founded recently and allows crypto projects to connect to reliable and trusted influencers and KOLs to drive engagement and improve marketability. The centralized platform establishes a mutually beneficial link between influencers and crypto investors that is mutually beneficial and transparent. The company uses advanced technology to match ideal influencer candidates to crypto projects and helps to source seed funding for start-up crypto projects. Contact Information Website: www.seedhunter.com
Email: contact@seedhunter.com
Twitter: @SeedHunterHQ
LinkedIn: SeedHunter Contact Chief Operating Officer Alexandre Barros SeedHunter contact@seedhunter.com This story was distributed as a release by Chainwire under HackerNoon’s Business Blogging Program. Learn more about the program here. DUBAI, UAE, April 18th, 2024/Chainwire/--SeedHunter, is a dedicated platform that facilitates and fosters collaborations between influencers and crypto projects to gain traction in the web3 community. The company has a new and fresh perspective on how to match high-potential influencers and KOLs with respective crypto projects and investors in a lucrative, scalable, and secure way. SeedHunter launched on April 18th 2024, its debut service: The Marketing Module. This is a platform that allows crypto projects to manage and maintain their influencer marketing campaigns more efficiently, securely, and effectively. The marketing campaigns launched can be uniquely customized to the crypto project’s needs in terms of content, deliverables, and more. The platform then uses an AI-based data model to evaluate, rate, and match reliable KOLs to crypto projects. SeedHunter will also assess influencer performance and define pricing directly using an advanced algorithm based on metrics of engagement, impression rates, number of followers, and the compatibility of the KOL with the campaign they manage - all of these standards will ensure that the right influencer is matched to the user's target audience needs. This ensures a fully transparent, fair, and lucrative influencer marketing position for influencers of all kinds. Crypto projects can also benefit from several built-in marketing campaign tools to drive engagement - including consultancy and account management as needed. SeedHunter takes care of all the tedious admin to provide a simplified, insightful, and efficient platform to view potential influencer and KOL partnerships. Dariusz Kowalczyk, the co-founder and CTO of SeedHunter, has stated that the platform “does the due diligence to find reliable and relevant KOLs that match your target audience’s interests. This saves time and energy while pulling the focus onto the customer.” Dariusz Kowalczyk, the co-founder and CTO of SeedHunter, has stated that the platform “does the due diligence to find reliable and relevant KOLs that match your target audience’s interests. This saves time and energy while pulling the focus onto the customer.” SeedHunter also makes cryptocurrency projects a lot more accessible by simplifying processes. Influencer campaigns can be created in a matter of minutes through a centralized platform - giving the community automated contracts, KOL reviews, and delivery checks. Payments will also be based on deliveries to ensure cost-effective marketing. The web3 community can easily use SeedHunter to curate the ideal influencer campaign for their project and push growth. To celebrate its official launch, SeedHunter is diving right in with a platform campaign open to all interested KOLs. This provides an opportunity for influencers to get involved from the get-go. SeedHunter has also three additional, high-profile projects waiting to be launched. These span the exciting realms of blockchain, entertainment, and gaming, offering KOLs a chance to contribute to cutting-edge initiatives and propel their marketing efforts. The crypto industry is uniquely disruptive and as such, it needs platforms like SeedHunter that offer scalable solutions from the very beginning for a lasting chance of success. With automated, simplified, and user-friendly interfaces, SeedHunter provides a seamless experience for influencers and crypto projects alike. Co-founder and CEO, Junaid Dar has said, “The SeedHunter platform will act as a canvas for prospective reliable influencers and KOLs to connect to crypto projects that are looking to gain traction in the market in a way that will simplify, secure, and inspire the industry.” Co-founder and CEO, Junaid Dar has said, “The SeedHunter platform will act as a canvas for prospective reliable influencers and KOLs to connect to crypto projects that are looking to gain traction in the market in a way that will simplify, secure, and inspire the industry.” SeedHunter is built on the values of trust, efficiency, collaboration, dynamics, and prosperity. This has helped the company establish a clear, guided vision for the future of the crypto influencer market. The platform’s ultimate mission is to ensure an easier, scalable, and more secure channel through which innovative crypto projects can be matched to specific influencers and investors in a way that adds value and clarity to a dynamic cryptocurrency market. SeedHunter invites crypto project managers and influencers of all sizes to take advantage of the Marketing Module that simplifies collaborations in a mutually beneficial, fair, and secure way. SeedHunter is also open to partnerships in the cryptocurrency industry that would help to expand and establish the vision of efficient, secure, and reliable collaborations across the web3 community. Moving forward, SeedHunter will also be launching its Funding Module in June 2024 which will allow crypto projects to find the necessary seed capital needed to gain traction. In this project, SeedHunter will act as a broker between the community and crypto projects that need funding. This project ensures the involvement of smaller influencers as well and allows the project to engage with communities and participate in projects for only $50. This ensures safety, transparency, and safety for start-up crypto projects looking to change the industry. About SeedHunter SeedHunter was founded recently and allows crypto projects to connect to reliable and trusted influencers and KOLs to drive engagement and improve marketability. The centralized platform establishes a mutually beneficial link between influencers and crypto investors that is mutually beneficial and transparent. The company uses advanced technology to match ideal influencer candidates to crypto projects and helps to source seed funding for start-up crypto projects. Contact Information Website: www.seedhunter.com Email: contact@seedhunter.com Twitter: @SeedHunterHQ LinkedIn: SeedHunter Website: www.seedhunter.com www.seedhunter.com www.seedhunter.com Email: contact@seedhunter.com contact@seedhunter.com contact@seedhunter.com Twitter: @SeedHunterHQ @SeedHunterHQ @SeedHunterHQ LinkedIn: SeedHunter SeedHunter SeedHunter Contact Chief Operating Officer Alexandre Barros SeedHunter contact@seedhunter.com This story was distributed as a release by Chainwire under HackerNoon’s Business Blogging Program. Learn more about the program here. This story was distributed as a release by Chainwire under HackerNoon’s Business Blogging Program. Learn more about the program here. here. here.

SeedHunter Marketing Module Is live - Web3 Influencer Campaigns With Payment In Stable Coins

How to Integrate React and AngularJS via Webpack Module Federation

Whether it's a React front-end app, a Go CLI tool, or a UI design project, there is always initial toil to figure out the optimal folder structure. As this initial decision influences a lot of flexibility, extensibility, self-explanation, and easy maintenance in our projects, it is key decision to ensure a smooth developer experience. When working with a new tool/technology/framework, our journey typically starts with reading official "getting started" handbook from their official website or even reading some articles on the same topic. We use these resources and start getting our hands dirty with hands-on experience, often using its structure as a foundation for more complex real-world projects. But these articles or tutorials are often serve us good in initial phases of the project, when the complexity is low. When we are solving complex problem involving multiple actors, the legibility and maintainability takes precedence. It becomes a daunting task to later refactor or sometimes rewrite everything from the scratch. To reduce this hassle and tackle this issue head on, I've distilled my Terraform experience into a CLI tool. This tool generates a battle-tested folder structure along with basic modules, allowing us to quickly hit the ground running. Structuring Terraform Folders Most companies and their ops teams find it cumbersome to manage multiple environments across multiple regions for their applications. We can structure our terraform folders as follows: Folder structure organized by Region Folder structure by Resources ( like AWS EC2, or Azure Functions etc) Folder structure on use case ( like front-end app, networking etc) Folder structure organized by Account Folder structure organized by environment and A Hybrid of all the above Given the above options it quite becomes confusing to the teams starting with terraform to decide how to structure their projects. Based on my experience here are my 2 cents on how to structure a terraform project: Create a modular style code with each module containing all the resources required to create for each use-case. These modules would serve as base blueprints which can be utilized across different environments. For ex: In case of AWS, The front-end module should consist of Cloud-front, S3 bucket, cloud-front origin access control, s3 policy bucket policy and s3 bucket public access block. Create a folder structure for each of the environments that you are deploying. This statement would be true, if the architecture across all the environments doesn't change and their deployment strategies does not change. Tfblueprintgen: A Terraform tool to generate folder structure and base blueprints Based on the above postulates, i have created a CLI tool called Tfblueprintgen, which generates the folder structure along with the modular working blocks to create AWS building blocks.  In terms of folder structure, the structure will look something like below. To run the tool download the both windows and Linux binaries from or you can build your own binary from .  Use the the binary ( if in Windows double-click to run Tfblueprintgen.exe or if it is Linux run the binary using ./Tfblueprintgen) here here As described in the image 1, the tool generates two things: A Parent folder which contains all the main terraform files ( , variables and )for each environment separated in their own folders. outputs.tf .tf main.tf A Module folder which contains all the different basic resources, segregated in their own separate folders. These modules can be leveraged within each of the environment folders, by calling those modules using module block and these can be applied using “terraform apply” With this setup, you can hit the ground up and running in no time. Feel free to add more ideas as issues and Stay tuned to project. Liked my content? Feel free to reach out to my for interesting content and productive discussions. LinkedIn

Tfblueprintgen: A Tool to Simplify Terraform Folder Setup and Provide Base Resource Modules

The os package in Python provides a wide range of functionalities for interacting with the operating system. One of the most commonly used aspects of this package is its ability to perform file and folder operations such as iteration, deletion, creation, moving, and opening. In this article, we'll explore how to utilize the os package for these tasks. Iterate Through All the Items Within a Folder You can use the os package in Python to iterate through all the items within a folder using the os.listdir() method. And you can use the os.path.join function to concatenate the folder path with each item in the folder to get the absolute path. Here's an example of how to do it: import os

# Replace 'folder_path' with the path of the folder you want to iterate through
folder_path = 'path_to_folder'

# Iterate through all items in the folder
for item in os.listdir(folder_path):
		print(item) 
		# Get the absolute path of the item
		item_path = os.path.join(folder_path, item) 
		# Display the absolute path of the item
		print(item_path) In this example, returns a list of all the items (files and subfolders) in the specified folder. You can then iterate through this list and perform any operations you need for each item. os.listdir(folder_path) Remember to replace with the actual path of the folder you want to iterate through. 'path_to_folder' Deleting Files and Folders The package provides methods to delete both files and folders. Here's how you can achieve this: os To Delete a File: import os
os.remove('file_path/file.txt') To Delete an Empty Folder: import os
os.rmdir('folder_path')  # Replace with the actual folder path To Delete a Non-Empty Folder and All Its Contents: import shutil
shutil.rmtree('folder_path')  # Replace with the actual folder path Remember to replace and with the actual paths of the file or folder you want to delete. Use these functions with caution as they will permanently delete the specified files or folders. 'file_path/file.txt' 'folder_path' Creating Folders Creating a new folder using the package is straightforward: os import os
os.mkdir('new_folder')  # Replace with the desired folder name Moving and Renaming Files/Folders The package allows us to move and rename files and folders as well. Here's an example of moving a file: os import os
os.rename('old_location/file.txt', 'new_location/file.txt') Opening Files and Folders You can also open a folder in the default file explorer using the package. For instance, to open a folder in the File Explorer (Windows): os import os
os.startfile('folder_path')  # Replace with the actual folder path Conclusion The package in Python offers powerful capabilities for file and folder operations, allowing developers to efficiently manage file system tasks within their applications. os By utilizing the methods provided by the package, you can seamlessly iterate, delete, create, move, and open files and folders in your Python programs. os Thank you for taking the time to explore data-related insights with me. I appreciate your engagement. If you find this information helpful, I invite you to follow me or connect with me on . Happy exploring!👋 LinkedIn Also published here

How to Use the os Module for High-Level File Operations Such as Iterating, Moving, and More

Module Design Pattern is a way to encapsulate and organize code in a self-containing module that can expose certain functionalities while keeping the rest of the code private. This helps prevent variable and function name clashes, improves code maintainability, and promotes the concept of separation of concerns. Here's an example of how you can implement the Module Design Pattern in JavaScript: // Module using the Module Design Pattern
var Module = (function() {
  // Private variable
  var privateVariable = "I am a private variable";

  // Private function
  var privateFunction = function privateFunction() {
    console.log("This is a private function");
  }

  // Public function
  var publicFunction = function() {
      console.log("This is a public function");
  }
  // Public interface
  return {
    publicFunction:publicFunction,
    // Public variable accessing private variable
    publicVariable: privateVariable
  };
})();

// Usage of the module
console.log(Module.publicVariable); // Output: "I am a private variable"
Module.publicFunction(); // Output: "This is a public function"

// Trying to access private members directly (will result in an error)
console.log(Module.privateVariable); // Output: undefined
Module.privateFunction(); // Output: Uncaught TypeError: MyModule.privateFunction is not a function In the above example, we've created a module called using an . Inside the IIFE, we define private variables and functions that are not accessible from outside the module. We then return an object containing the public members (functions and variables) that we want to expose. Module Immediately Invoked Function Expression (IIFE) Benefits of the Module Design Pattern: Private members are not accessible from outside the module, which helps prevent unintended modifications or conflicts with other parts of the codebase. Encapsulation: Code is organized into logical modules, making it easier to manage and understand. Organization: The module acts as a namespace, reducing the chances of naming collisions in a global scope. Namespacing: Modules can be easily reused in different parts of your application or different projects. Reusability: Separation of concerns and encapsulation make it easier to maintain and refactor code. Maintenance: That's a concise explanation of the module design pattern in Javascript. Feel free to comment on how you like my blog or shoot me an email at If you have any queries, and I will try to answer. connect@nandan.dev You can also visit my website to read some of the articles at nandan.dev Stay tuned & connect with me on my social media channels. Make sure to subscribe to my newsletter to get regular updates on my upcoming posts. | | | Twitter Instagram Github Website Also published . here

A Quick Module Design Pattern Example in JavaScript

Stories, News & Mentions about MODULIZE

Modulize’s Stories on HackerNoon