Inizio

inizio

(1) Sean M. Stafford, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA;
(2) Alexander Aduenko, Moscow Institute of Physics and Technology, Moscow, Russia;
(3) Marcus Djokic, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA;
(4) Yu-Hsiu Lin, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA;
(5) Jose L. Mendoza-Cortes, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA (Email: jmendoza@msu.edu). Table of Links Abstract and Introduction SALSA- (S)ubstitution, (A)pproximation, Evo(L)utionary (S)earch, and (A)B-Initio Calculations SALSA Applied to Photocatalytic Water-splitting Discussion Methods Conclusions, Data Availability Statement and References Appendix: Supplementary Material II. SALSA – (S)UBSTITUTION, (A)PPROXIMATION, EVO(L)UTIONARY (S)EARCH, AND (A)B-INITIO CALCULATIONS We developed a highly efficient and versatile materials discovery process, dubbed SALSA, which is an acronym for Substitution, Approximation, evoLutionary Search, and Abinitio calculations. An overview of SALSA is provided in Figure 1. The process starts by taking a target property or set of properties as input and returns a set of candidate structures as output. Instead of relying on brute-force approaches, SALSA harnesses the power of a large database of compounds with known structures and properties to rapidly search for new materials. The process begins with swapping ionic components between pairs of known compounds that have similar ionic species, as guided by a substitution likelihood matrix, to produce a dataset of hybrid compounds with defined compositions but undefined structures. We then infer approximate properties for these hybrid compounds using a weighted sum of properties of parent compounds and discard hybrids without desirable properties. Promising hybrids are then subjected to an evolutionary structure search using the USPEX algorithm, which generates stable crystal structures for a given composition whenever possible. High-fidelity DFT calculations are then used to recalculate the properties of the generated structures, and structures with undesirable properties are discarded. The process produces a set of undiscovered materials that are promising candidates for various applications, including the application to artificial photosynthesis discussed in Section III. Furthermore, SALSA is highly versatile and can be applied to other materials science problems as well. a. Substitution by Chemical Similarity Our group reconstructed and expanded the scope of the substitution likelihood matrix introduced by Hautier et al. 19 In our construction, we used the entirety of the Inorganic Crystal Structure Database (ICSD)23 and do not restrict substitutions to preserve the space group of the crystal structure (Stafford et al., 2023b in prep will describe details of this construction.) High values of our matrix correspond to pairs of ionic species empirically observed to exist in similar chemical environments. Above a chosen threshold, a value designates substitution between an ion pair as likely. Applying these likely substitutions to compounds of our initial dataset forms a hypothetical set of new candidate compounds. The resulting candidate dataset is too large for us to feasibly calculate properties of all compounds unless we are overly restrictive with unit cell size or substitution threshold. Therefore, we narrow the scope of our investigation to a subset for which we can efficiently approximate properties. b. Approximation by Linear Interpolation We examine the class of candidate compounds which are compositional interpolations between two initial compounds, i.e. hybrid compounds. We derive estimates for the properties of hybrids by summing the properties of parent compounds with the same ratio used in the corresponding hybrid composition. Next, we define the boundary of a target region of property space appropriate for our application. Finally, we eliminate hybrids that do not lie within this region. This step allows us to filter out the sizeable portion of our candidate compounds that are far removed from the target region before proceeding to intensive calculations. While this is an extremely simplistic model of property space, it is a computationally cheap way to approximate values close enough to eliminate most of the unsuitable candidates without a high risk of eliminating suitable ones. Note that we reduce this risk by extending the boundary of our target region beyond the ideal region of property space by enough to include some tolerance for the error that comes with our interpolation method. See Figure 2 for a summary of this scheme. c. Evolutionary Search of Structure Space Until this point, we have defined our hybrid compounds by their composition alone, but reliable property calculations require structural information. Crystal structure prediction from first principles is prohibitively difficult using just composition. Instead, we turn to an evolutionary structure search code, USPEX, to generate crystal structures for our hybrids. We provide USPEX with a hybrid composition and enable all available stochastic variation operations, which includes variation of the space group. If USPEX is unable to converge a structure for a given composition, that indicates the composition is unlikely to have a thermodynamically stable structure and is eliminated from further consideration. See Section V E for a more detailed look at our USPEX methodology. d. Ab-initio Property Calculations Our candidate set is now vastly narrowed down and contains structural information so high fidelity property calculations are computationally feasible. Therefore we perform geometry optimization and property calculation with another DFT code, CRYSTAL17, at the hybrid functional level of theory.24,25 Some candidate compounds located within the target region according to interpolation-inferred values shift outside the region upon replacement by CRYSTAL17-calculated values while others do not converge with CRYSTAL17 at all. We discard these and are left with the final products of SALSA – the structures which CRYSTAL17 converges and determines to have properties in the target region. This paper is available on arxiv under CC 4.0 license. (1) Sean M. Stafford, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA; (2) Alexander Aduenko, Moscow Institute of Physics and Technology, Moscow, Russia; (3) Marcus Djokic, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA; (4) Yu-Hsiu Lin, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA; (5) Jose L. Mendoza-Cortes, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA (Email: jmendoza@msu.edu). (1) Sean M. Stafford, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA; (2) Alexander Aduenko, Moscow Institute of Physics and Technology, Moscow, Russia; (3) Marcus Djokic, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA; (4) Yu-Hsiu Lin, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA; (5) Jose L. Mendoza-Cortes, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA (Email: jmendoza@msu.edu). Table of Links Abstract and Introduction Abstract and Introduction SALSA- (S)ubstitution, (A)pproximation, Evo(L)utionary (S)earch, and (A)B-Initio Calculations SALSA- (S)ubstitution, (A)pproximation, Evo(L)utionary (S)earch, and (A)B-Initio Calculations SALSA Applied to Photocatalytic Water-splitting SALSA Applied to Photocatalytic Water-splitting Discussion Discussion Methods Methods Conclusions, Data Availability Statement and References Conclusions, Data Availability Statement and References Appendix: Supplementary Material Appendix: Supplementary Material II. SALSA – (S)UBSTITUTION, (A)PPROXIMATION, EVO(L)UTIONARY (S)EARCH, AND (A)B-INITIO CALCULATIONS We developed a highly efficient and versatile materials discovery process, dubbed SALSA, which is an acronym for Substitution, Approximation, evoLutionary Search, and Abinitio calculations. An overview of SALSA is provided in Figure 1. The process starts by taking a target property or set of properties as input and returns a set of candidate structures as output. Instead of relying on brute-force approaches, SALSA harnesses the power of a large database of compounds with known structures and properties to rapidly search for new materials. The process begins with swapping ionic components between pairs of known compounds that have similar ionic species, as guided by a substitution likelihood matrix, to produce a dataset of hybrid compounds with defined compositions but undefined structures. We then infer approximate properties for these hybrid compounds using a weighted sum of properties of parent compounds and discard hybrids without desirable properties. Promising hybrids are then subjected to an evolutionary structure search using the USPEX algorithm, which generates stable crystal structures for a given composition whenever possible. High-fidelity DFT calculations are then used to recalculate the properties of the generated structures, and structures with undesirable properties are discarded. The process produces a set of undiscovered materials that are promising candidates for various applications, including the application to artificial photosynthesis discussed in Section III. Furthermore, SALSA is highly versatile and can be applied to other materials science problems as well. a. Substitution by Chemical Similarity Our group reconstructed and expanded the scope of the substitution likelihood matrix introduced by Hautier et al. 19 In our construction, we used the entirety of the Inorganic Crystal Structure Database (ICSD)23 and do not restrict substitutions to preserve the space group of the crystal structure (Stafford et al., 2023b in prep will describe details of this construction.) High values of our matrix correspond to pairs of ionic species empirically observed to exist in similar chemical environments. Above a chosen threshold, a value designates substitution between an ion pair as likely. Applying these likely substitutions to compounds of our initial dataset forms a hypothetical set of new candidate compounds. The resulting candidate dataset is too large for us to feasibly calculate properties of all compounds unless we are overly restrictive with unit cell size or substitution threshold. Therefore, we narrow the scope of our investigation to a subset for which we can efficiently approximate properties. b. Approximation by Linear Interpolation We examine the class of candidate compounds which are compositional interpolations between two initial compounds, i.e. hybrid compounds. We derive estimates for the properties of hybrids by summing the properties of parent compounds with the same ratio used in the corresponding hybrid composition. Next, we define the boundary of a target region of property space appropriate for our application. Finally, we eliminate hybrids that do not lie within this region. This step allows us to filter out the sizeable portion of our candidate compounds that are far removed from the target region before proceeding to intensive calculations. While this is an extremely simplistic model of property space, it is a computationally cheap way to approximate values close enough to eliminate most of the unsuitable candidates without a high risk of eliminating suitable ones. Note that we reduce this risk by extending the boundary of our target region beyond the ideal region of property space by enough to include some tolerance for the error that comes with our interpolation method. See Figure 2 for a summary of this scheme. c. Evolutionary Search of Structure Space Until this point, we have defined our hybrid compounds by their composition alone, but reliable property calculations require structural information. Crystal structure prediction from first principles is prohibitively difficult using just composition. Instead, we turn to an evolutionary structure search code, USPEX, to generate crystal structures for our hybrids. We provide USPEX with a hybrid composition and enable all available stochastic variation operations, which includes variation of the space group. If USPEX is unable to converge a structure for a given composition, that indicates the composition is unlikely to have a thermodynamically stable structure and is eliminated from further consideration. See Section V E for a more detailed look at our USPEX methodology. d. Ab-initio Property Calculations Our candidate set is now vastly narrowed down and contains structural information so high fidelity property calculations are computationally feasible. Therefore we perform geometry optimization and property calculation with another DFT code, CRYSTAL17, at the hybrid functional level of theory.24,25 Some candidate compounds located within the target region according to interpolation-inferred values shift outside the region upon replacement by CRYSTAL17-calculated values while others do not converge with CRYSTAL17 at all. We discard these and are left with the final products of SALSA – the structures which CRYSTAL17 converges and determines to have properties in the target region. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

SALSA- (S)ubstitution, (A)pproximation, Evo(L)utionary (S)earch, and (A)B-Initio Calculations

(1) Sean M. Stafford, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA;
(2) Alexander Aduenko, Moscow Institute of Physics and Technology, Moscow, Russia;
(3) Marcus Djokic, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA;
(4) Yu-Hsiu Lin, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA;
(5) Jose L. Mendoza-Cortes, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA (Email: jmendoza@msu.edu). Table of Links Abstract and Introduction SALSA- (S)ubstitution, (A)pproximation, Evo(L)utionary (S)earch, and (A)B-Initio Calculations SALSA Applied to Photocatalytic Water-splitting Discussion Methods Conclusions, Data Availability Statement and References Appendix: Supplementary Material Appendix: Supplementary Material 1. Hybrid Compound Enumeration and Max Unit Cell Size 2. Interpolation Distance Analysis The four highest φox compounds with low-Eg and high-Eg, respectively are Ag2Te, Ag2Se, PbSe, and PbTe; and AgBr, TiO2, AgCl, and CuCl, ordered by their total interpolation distance. These eight account for nearly 70% of interpolation distance. SnO2, Cu2S, and PbS also make significant, albeit lesser contributions to interpolation. 3. Miscellaneous Additional Candidate Compound Data 4. Price Calculation We estimated a price (USD/kg) for final structures in Table III by performing a weighted sum of component elements prices. The component elements prices were weighted by fractional composition of the final structures. These elemental prices were found in technical reports.27–31 5. Initial Compound Distribution 6. CIFs for Optimized Final Structures This paper is available on arxiv under CC 4.0 license. (1) Sean M. Stafford, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA; (2) Alexander Aduenko, Moscow Institute of Physics and Technology, Moscow, Russia; (3) Marcus Djokic, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA; (4) Yu-Hsiu Lin, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA; (5) Jose L. Mendoza-Cortes, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA (Email: jmendoza@msu.edu). (1) Sean M. Stafford, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA; (2) Alexander Aduenko, Moscow Institute of Physics and Technology, Moscow, Russia; (3) Marcus Djokic, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA; (4) Yu-Hsiu Lin, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA; (5) Jose L. Mendoza-Cortes, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA (Email: jmendoza@msu.edu). Table of Links Abstract and Introduction SALSA- (S)ubstitution, (A)pproximation, Evo(L)utionary (S)earch, and (A)B-Initio Calculations SALSA Applied to Photocatalytic Water-splitting Discussion Methods Conclusions, Data Availability Statement and References Appendix: Supplementary Material Appendix: Supplementary Material 1. Hybrid Compound Enumeration and Max Unit Cell Size 2. Interpolation Distance Analysis The four highest φox compounds with low-Eg and high-Eg, respectively are Ag2Te, Ag2Se, PbSe, and PbTe; and AgBr, TiO2, AgCl, and CuCl, ordered by their total interpolation distance. These eight account for nearly 70% of interpolation distance. SnO2, Cu2S, and PbS also make significant, albeit lesser contributions to interpolation. 3. Miscellaneous Additional Candidate Compound Data 4. Price Calculation We estimated a price (USD/kg) for final structures in Table III by performing a weighted sum of component elements prices. The component elements prices were weighted by fractional composition of the final structures. These elemental prices were found in technical reports.27–31 5. Initial Compound Distribution 6. CIFs for Optimized Final Structures This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Analyzing Interpolation Distances in Photocatalytic Materials

Advances in Photocatalytic Materials through SALSA

(1) Sean M. Stafford, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA;
(2) Alexander Aduenko, Moscow Institute of Physics and Technology, Moscow, Russia;
(3) Marcus Djokic, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA;
(4) Yu-Hsiu Lin, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA;
(5) Jose L. Mendoza-Cortes, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA (Email: jmendoza@msu.edu). Table of Links Abstract and Introduction SALSA- (S)ubstitution, (A)pproximation, Evo(L)utionary (S)earch, and (A)B-Initio Calculations SALSA Applied to Photocatalytic Water-splitting Discussion Methods Conclusions, Data Availability Statement and References Appendix: Supplementary Material V. METHODS A. Initial Dataset We first collected a dataset of experimentally determined semiconductor band gaps. We then applied the method described in Stafford et al. 2023c in prep to calculate reduction and oxidation potentials. This formed an initial dataset containing 34 compounds. We sought compounds with band gaps between 1.23–2.80 eV to enable efficient photoabsorption. We also sought reduction potentials below 0.00 V and oxidation potentials above 1.23 V, with respect to the NHE, for materials which are stable in an aqueous environment. None of our original materials had suitable values for all three properties. Figure 7 presents an overview of the collection of initial compounds and the region of property space described above. Table IV lists each initial compound, its band gap and its redox potentials. Figure 8 in the Supplementary Material presents a closer look at the swarm of compounds that hover just outside of the target space. Few compounds are stable to photo-catalyzed decomposition. This is mainly because most have oxidation potentials that are too low, leaving them prone to hole oxidation (Figure 8 (d)). However, some of the few with acceptable oxidation potentials have reduction potentials that are too high (Figure 8 (b)). Additionally, compounds are roughly evenly divided into three groups which cannot absorb sunlight efficiently, cannot absorb it at all in the regions of higher solar intensity, and which have an acceptable band gap (Figure 8 (e) and (c)). No matter which angle we look at this property space, we see there is great room for improvement. B. Parameters Used for Candidate Generation a. Substitution Threshold We used a substitution threshold of 0, that is, we did not consider substitutions associated with negative values in our substitution likelihood matrix. This parameter can be adjusted as governed by the computational resources available to a search. A lower threshold enables a more thorough exploration of composition space, but is more computationally expensive and less efficient at finding suitable materials. b. Substitution Implementation We allowed substitution to constitute a complete or partial replacement of the original ion. For example, Br− ↔ I − is a matrix-allowed substitution and AgBr is in our initial dataset so compounds of the form AgnBrn –m Im with n,m ∈ Z, are in our candidate dataset. We limited substitutions to be first or second order, i.e at most two substitutions could be used to generate an individual candidate. In Section III, first and second-order substitutions correspond to ternary and quaternary compounds, respectively. Theoretically, a second-order substitution could consist of exchanging a single, original ionic component for two new ions. However, second-order substitutions that formed hybrid compounds consisted of a single substitution of each of the original components as this is the only way a secondorder substitution could correspond to interpolation between two binary compounds. Building on the previous example, the substitution Ag+ ↔ Cu+ could be used in second-order substitutions to produce quaternary compounds of the form Agn – pCupBrn –m Im with n, p,m ∈ Z. For the purpose of enumerating a complete dataset, the new components of candidate compounds were limited to half or less the final composition. So all second-order substitutions were partial and you could not generate CuI (n = p = m = 1) from just AgBr. However, this limitation does not affect hybrid compounds. c. Unit Cell Size Limit In practice, the enumeration of candidate compounds requires some constraint on values of n,m, p. Results presented in Section III implemented this constraint by imposing a maximum of 20 atoms in a unit cell. This is equivalent to the constraints 1 ≤ n, p,m ≤ 10 in the previous example. C. Property Space Selection Criteria With our interpolation scheme we filtered compounds that did not meet the following criteria: 1.03 < band gap (eV) < 3.00, oxidation potential (V) > 1.03, and reduction potential (V) < 0.2. This includes an extra window of 0.20 eV for the band gaps and 0.20 V for the potentials to allow for materials that might ultimately arrive in the desired region of property space by deviating slightly from their linear interpolation. To illustrate this process, consider PbSe and CuCl. PbSe’s band gap is too small, at 0.27 eV, and its oxidation potential is too low, at 0.76 eV, while CuCl has too high a band gap at 3.40 eV. However, the 50:50 interpolation between these two, PbCuSeCl, has band gap, oxidation and reduction potentials of 1.84 eV, 1.23 V and −0.25 V, respectively, which places it just inside the threshold of our target region. D. Interpolation We construct hybrid compositions which are integer ratios of two parent compositions. We then estimate the properties of the corresponding hybrid compounds to be linear interpolations of the parent compounds on a per-atom basis. In other words, we weight the initial property values by the number of atoms contributed to the hybrid. Furthermore, we don’t restrict our interpolations to be single-substitution. For example, both Pb2+ ↔ Cu+ and Se2− ↔ Cl− are matrixallowed substitutions so if we start with initial compounds PbSe and CuCl, we generate interpolated compositions such as PbCuSeCl. E. USPEX Settings We provide USPEX a composition and allow it to perform all stochastic modifications it has at its disposal. We do not constrain the structure by space group. For energy evaluation, we elect USPEX’s option to interface with the DFT code, Vienna Ab initio Simulation Package (VASP).32–34 All VASP calculations were performed in the plane-wave DFT framework at the Generalized Gradient Approximation (GGA) level of theory and used the Perdew, Burke, and Ernzerhof (PBE) functional.35 Projector-augmented wave (PAW) pseudopotentials were used to represent the core electrons and ion-electron interactions.36,37 We used a plane-wave cutoff of 500 eV, an energy convergence criterion of 10−4 eV, and force convergence of 0.02 eV/A. Dispersive interactions were ac- ˚ counted for using DFT-D3 corrections38 with Becke-Jonson damping.39 We also included spin polarization effects. F. CRYSTAL17 Settings This paper is available on arxiv under CC 4.0 license. (1) Sean M. Stafford, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA; (2) Alexander Aduenko, Moscow Institute of Physics and Technology, Moscow, Russia; (3) Marcus Djokic, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA; (4) Yu-Hsiu Lin, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA; (5) Jose L. Mendoza-Cortes, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA (Email: jmendoza@msu.edu). (1) Sean M. Stafford, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA; (2) Alexander Aduenko, Moscow Institute of Physics and Technology, Moscow, Russia; (3) Marcus Djokic, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA; (4) Yu-Hsiu Lin, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA; (5) Jose L. Mendoza-Cortes, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA (Email: jmendoza@msu.edu). Table of Links Abstract and Introduction Abstract and Introduction SALSA- (S)ubstitution, (A)pproximation, Evo(L)utionary (S)earch, and (A)B-Initio Calculations SALSA- (S)ubstitution, (A)pproximation, Evo(L)utionary (S)earch, and (A)B-Initio Calculations SALSA Applied to Photocatalytic Water-splitting SALSA Applied to Photocatalytic Water-splitting Discussion Discussion Methods Methods Conclusions, Data Availability Statement and References Conclusions, Data Availability Statement and References Appendix: Supplementary Material Appendix: Supplementary Material V. METHODS A. Initial Dataset We first collected a dataset of experimentally determined semiconductor band gaps. We then applied the method described in Stafford et al. 2023c in prep to calculate reduction and oxidation potentials. This formed an initial dataset containing 34 compounds. We sought compounds with band gaps between 1.23–2.80 eV to enable efficient photoabsorption. We also sought reduction potentials below 0.00 V and oxidation potentials above 1.23 V, with respect to the NHE, for materials which are stable in an aqueous environment. None of our original materials had suitable values for all three properties. Figure 7 presents an overview of the collection of initial compounds and the region of property space described above. Table IV lists each initial compound, its band gap and its redox potentials. Figure 8 in the Supplementary Material presents a closer look at the swarm of compounds that hover just outside of the target space. Few compounds are stable to photo-catalyzed decomposition. This is mainly because most have oxidation potentials that are too low, leaving them prone to hole oxidation (Figure 8 (d)). However, some of the few with acceptable oxidation potentials have reduction potentials that are too high (Figure 8 (b)). Additionally, compounds are roughly evenly divided into three groups which cannot absorb sunlight efficiently, cannot absorb it at all in the regions of higher solar intensity, and which have an acceptable band gap (Figure 8 (e) and (c)). No matter which angle we look at this property space, we see there is great room for improvement. B. Parameters Used for Candidate Generation a. Substitution Threshold We used a substitution threshold of 0, that is, we did not consider substitutions associated with negative values in our substitution likelihood matrix. This parameter can be adjusted as governed by the computational resources available to a search. A lower threshold enables a more thorough exploration of composition space, but is more computationally expensive and less efficient at finding suitable materials. b. Substitution Implementation We allowed substitution to constitute a complete or partial replacement of the original ion. For example, Br− ↔ I − is a matrix-allowed substitution and AgBr is in our initial dataset so compounds of the form AgnBrn –m Im with n,m ∈ Z, are in our candidate dataset. We limited substitutions to be first or second order, i.e at most two substitutions could be used to generate an individual candidate. In Section III, first and second-order substitutions correspond to ternary and quaternary compounds, respectively. Theoretically, a second-order substitution could consist of exchanging a single, original ionic component for two new ions. However, second-order substitutions that formed hybrid compounds consisted of a single substitution of each of the original components as this is the only way a secondorder substitution could correspond to interpolation between two binary compounds. Building on the previous example, the substitution Ag+ ↔ Cu+ could be used in second-order substitutions to produce quaternary compounds of the form Agn – pCupBrn –m Im with n, p,m ∈ Z. For the purpose of enumerating a complete dataset, the new components of candidate compounds were limited to half or less the final composition. So all second-order substitutions were partial and you could not generate CuI (n = p = m = 1) from just AgBr. However, this limitation does not affect hybrid compounds. c. Unit Cell Size Limit In practice, the enumeration of candidate compounds requires some constraint on values of n,m, p. Results presented in Section III implemented this constraint by imposing a maximum of 20 atoms in a unit cell. This is equivalent to the constraints 1 ≤ n, p,m ≤ 10 in the previous example. C. Property Space Selection Criteria With our interpolation scheme we filtered compounds that did not meet the following criteria: 1.03 < band gap (eV) < 3.00, oxidation potential (V) > 1.03, and reduction potential (V) < 0.2. This includes an extra window of 0.20 eV for the band gaps and 0.20 V for the potentials to allow for materials that might ultimately arrive in the desired region of property space by deviating slightly from their linear interpolation. To illustrate this process, consider PbSe and CuCl. PbSe’s band gap is too small, at 0.27 eV, and its oxidation potential is too low, at 0.76 eV, while CuCl has too high a band gap at 3.40 eV. However, the 50:50 interpolation between these two, PbCuSeCl, has band gap, oxidation and reduction potentials of 1.84 eV, 1.23 V and −0.25 V, respectively, which places it just inside the threshold of our target region. D. Interpolation We construct hybrid compositions which are integer ratios of two parent compositions. We then estimate the properties of the corresponding hybrid compounds to be linear interpolations of the parent compounds on a per-atom basis. In other words, we weight the initial property values by the number of atoms contributed to the hybrid. Furthermore, we don’t restrict our interpolations to be single-substitution. For example, both Pb2+ ↔ Cu+ and Se2− ↔ Cl− are matrixallowed substitutions so if we start with initial compounds PbSe and CuCl, we generate interpolated compositions such as PbCuSeCl. E. USPEX Settings We provide USPEX a composition and allow it to perform all stochastic modifications it has at its disposal. We do not constrain the structure by space group. For energy evaluation, we elect USPEX’s option to interface with the DFT code, Vienna Ab initio Simulation Package (VASP).32–34 All VASP calculations were performed in the plane-wave DFT framework at the Generalized Gradient Approximation (GGA) level of theory and used the Perdew, Burke, and Ernzerhof (PBE) functional.35 Projector-augmented wave (PAW) pseudopotentials were used to represent the core electrons and ion-electron interactions.36,37 We used a plane-wave cutoff of 500 eV, an energy convergence criterion of 10−4 eV, and force convergence of 0.02 eV/A. Dispersive interactions were ac- ˚ counted for using DFT-D3 corrections38 with Becke-Jonson damping.39 We also included spin polarization effects. F. CRYSTAL17 Settings This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

The SALSA Method in Photocatalytic Material Discovery

(1) Sean M. Stafford, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA;
(2) Alexander Aduenko, Moscow Institute of Physics and Technology, Moscow, Russia;
(3) Marcus Djokic, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA;
(4) Yu-Hsiu Lin, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA;
(5) Jose L. Mendoza-Cortes, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA (Email: jmendoza@msu.edu). Table of Links Abstract and Introduction SALSA- (S)ubstitution, (A)pproximation, Evo(L)utionary (S)earch, and (A)B-Initio Calculations SALSA Applied to Photocatalytic Water-splitting Discussion Methods Conclusions, Data Availability Statement and References Appendix: Supplementary Material IV. DISCUSSION Figure 6 (a) presents the interpolations into property space from our initial compounds which yielded our final structures, as well as shifts from interpolated predictions. Trends within this subset of interpolations suggest certain paths are favored for producing a photocatalytic water-splitter. Our final materials can be divided into two groups. One group is made up of materials containing Silver, halides and group 16 compounds. Among the few compounds in our initial dataset which had good oxidation potentials, most contained Silver, so this group emerged from the interpolation between a pair of materials which had good oxidation potentials, but which had band gaps that were too low and too high respectively. Consequently, these materials are robust to hole oxidation – all interpolated oxidation potentials are at least 0.2 V greater than the ideal minimum. However, their interpolated reduction potentials lie close to the threshold for rejection – none are more than 0.01 V under the ideal maximum. Additionally, these structures have low symmetry and are expensive due to their Silver content. The other group contains Lead instead of Silver. Redox suitability of these Lead compounds is inverted relative to the Silver group. That is, these compounds are robust to electron reduction due to Lead’s especially negative reduction potential – all have reduction potentials more than 0.2 V under the ideal maximum. However, none have an interpolated oxidation potential that is 0.03 V greater than the ideal minimum. The Lead structures are also higher in symmetry and relatively cheap. Figure 6 (b) highlights how compounds from different groups lie near different planes of the desired property space, demonstrating the strengths and weaknesses of these groups. The Lead group is about 50 times cheaper so it may offer more scalability.27–31 However, the Silver group follows more closely to regular compositional formula. This means it may be easier to find more compounds in this group with different interpolation ratios if the ones we have discovered do not prove to be as effective as they appear to be. Both paths should be investigated experimentally. Furthermore, we envision the materials design approach we used to be generalizable. In a different scheme, we would see a picture similar to this, but with different starting compounds and different boundaries than our target property space. This paper is available on arxiv under CC 4.0 license. (1) Sean M. Stafford, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA; (2) Alexander Aduenko, Moscow Institute of Physics and Technology, Moscow, Russia; (3) Marcus Djokic, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA; (4) Yu-Hsiu Lin, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA; (5) Jose L. Mendoza-Cortes, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA (Email: jmendoza@msu.edu). (1) Sean M. Stafford, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA; (2) Alexander Aduenko, Moscow Institute of Physics and Technology, Moscow, Russia; (3) Marcus Djokic, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA; (4) Yu-Hsiu Lin, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA; (5) Jose L. Mendoza-Cortes, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA (Email: jmendoza@msu.edu). Table of Links Abstract and Introduction Abstract and Introduction SALSA- (S)ubstitution, (A)pproximation, Evo(L)utionary (S)earch, and (A)B-Initio Calculations SALSA- (S)ubstitution, (A)pproximation, Evo(L)utionary (S)earch, and (A)B-Initio Calculations SALSA Applied to Photocatalytic Water-splitting SALSA Applied to Photocatalytic Water-splitting Discussion Discussion Methods Methods Conclusions, Data Availability Statement and References Conclusions, Data Availability Statement and References Appendix: Supplementary Material Appendix: Supplementary Material IV. DISCUSSION Figure 6 (a) presents the interpolations into property space from our initial compounds which yielded our final structures, as well as shifts from interpolated predictions. Trends within this subset of interpolations suggest certain paths are favored for producing a photocatalytic water-splitter. Our final materials can be divided into two groups. One group is made up of materials containing Silver, halides and group 16 compounds. Among the few compounds in our initial dataset which had good oxidation potentials, most contained Silver, so this group emerged from the interpolation between a pair of materials which had good oxidation potentials, but which had band gaps that were too low and too high respectively. Consequently, these materials are robust to hole oxidation – all interpolated oxidation potentials are at least 0.2 V greater than the ideal minimum. However, their interpolated reduction potentials lie close to the threshold for rejection – none are more than 0.01 V under the ideal maximum. Additionally, these structures have low symmetry and are expensive due to their Silver content. The other group contains Lead instead of Silver. Redox suitability of these Lead compounds is inverted relative to the Silver group. That is, these compounds are robust to electron reduction due to Lead’s especially negative reduction potential – all have reduction potentials more than 0.2 V under the ideal maximum. However, none have an interpolated oxidation potential that is 0.03 V greater than the ideal minimum. The Lead structures are also higher in symmetry and relatively cheap. Figure 6 (b) highlights how compounds from different groups lie near different planes of the desired property space, demonstrating the strengths and weaknesses of these groups. The Lead group is about 50 times cheaper so it may offer more scalability.27–31 However, the Silver group follows more closely to regular compositional formula. This means it may be easier to find more compounds in this group with different interpolation ratios if the ones we have discovered do not prove to be as effective as they appear to be. Both paths should be investigated experimentally. Furthermore, we envision the materials design approach we used to be generalizable. In a different scheme, we would see a picture similar to this, but with different starting compounds and different boundaries than our target property space. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Evaluating the Economic Viability of Silver and Lead-Based Photocatalysts

(1) Sean M. Stafford, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA;
(2) Alexander Aduenko, Moscow Institute of Physics and Technology, Moscow, Russia;
(3) Marcus Djokic, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA;
(4) Yu-Hsiu Lin, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA;
(5) Jose L. Mendoza-Cortes, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA (Email: jmendoza@msu.edu). Table of Links Abstract and Introduction SALSA- (S)ubstitution, (A)pproximation, Evo(L)utionary (S)earch, and (A)B-Initio Calculations SALSA Applied to Photocatalytic Water-splitting Discussion Methods Conclusions, Data Availability Statement and References Appendix: Supplementary Material III. SALSA APPLIED TO PHOTOCATALYTIC WATER-SPLITTING We found that millions of candidate compounds could be generated from our initial dataset with the ion exchanges suggested by our substitution matrix. Of these, about 13,600 were compatible with our structural interpolation scheme, that is, they could be constructed as hybrids of compounds within our initial dataset of known semiconductors. See Section V B for details on this dataset construction. A. Candidate Compounds Overall, we found about 1250 hybrid compounds within our target region, including 484 within our ideal region. This corresponds to roughly one out of every 10 and 30 of all possible hybrids, respectively. Most interpolation pairings involved binary compounds with no elements in common so more hybrids were quaternary rather than ternary. Furthermore, the binary parents of ternary compounds tended to be located more closely to each other in property space, without any portion of the target region between them, so ternary compounds were relatively underrepresented in the regions of interest. The quaternary:ternary ratio was about 5:1 overall, 7:1 in the target region, and 8:1 in the ideal region. Figure 3 provides insight into how certain interpolation patterns emerged as dominant. These patterns can be understood in relation to the initial distribution of compounds in property space. Few initial compounds had acceptable Eg or φox and none had both simultaneously; however, acceptable φred was much more common. This combination advantageously positioned those with relatively high φox, especially the circled cluster containing the five highest φox compounds, because many partners were located across the ideal region from them. In fact, compounds from this cluster constituted one partner in nearly all interpolation pairings depicted in Figure 3, with the other partner being out-of-cluster and usually low φred. These pairings had the largest interpolation distance within the ideal region when the out-of-cluster partner was among the highest φox of the low-Eg compounds. Larger interpolation distance correlates with a greater number of possible hybrid compounds so this was the most dominant type of interpolation in our hybrid compound dataset. Thus we can roughly understand the interpolation opportunities available to our dataset by focusing on just a small subset of low-Eg and high-Eg compounds which are least oxidizable. The four highest φox compounds in the high-Eg cluster were AgBr, TiO2, AgCl, and CuCl, ordered by the number of hybrids derived from them. 95% of hybrid compounds had a parent in this group, including 42% from AgBr alone. The four highest φox compounds with low-Eg were the binary combinations of Pb and Ag with Se and Te. 40% of hybrids had a parent in this group and 36% that had one parent from each group. Table I provides some example hybrid compounds from the target region including three hybrids of different composition from pairs of AgBr and TiO2 with lower Eg compounds. The variety of hybrids included represents how different parents produced hybrids in different regions, e.g. TiO2 – PbO hybrids tended to have low Eg. B. Candidate Structures We used the procedure for USPEX and VASP laid out in Section V E to search for the crystal structures of hybrids in our target region. USPEX was able to converge structures for about 50 hybrid compounds. The elemental composition of these structures mostly coincides with the composition in the hybrid compounds highlighted in the previous Section. For example, Ag has the greatest occurrence by far, due to its presence in both the low and high Eg groups. However, Br has a surprisingly much lower occurrence and Cd has a relatively higher occurrence. Figure 4 and Table II show example results of USPEX converged structures. Figure 4 also connects shifts in composition to changes in structure and property space. USPEX allowed us to find hybrid structures that had different space groups from their parent structures. For example, PbCuSeCl has a space group of 156, which is different than any known PbSe or CuCl structure according to AFLOW.26 The approach of Hautier et al. 19 and most other materials searches using the substitution matrix technique would not have allowed the discovery of this structure because they enforced a restriction new candidate compounds had the same crystal structure as compounds from which they were derived. C. Final Water-Splitters Finally, we used the hybrid DFT code, CRYSTAL17, to conduct higher fidelity geometry optimization on our candidate structures. Many of the USPEX-converged structures had to be eliminated because the band gap derived from CRYSTAL17 moved them out of the target region of property space and others had to be eliminated because they would not converge. Structures containing Ag had larger discrepancies between their interpolated Eg and their CRYSTAL17-calculated Eg. On average the difference was 0.92 eV compared to 0.51 eV for non-Ag compounds. The largest difference was for Ag4Cl2Se which decreased by 1.17 eV, placing it slightly out of the ideal region. The surviving structures mostly had low symmetry. Figure 5 and Table III provide examples of final structures that remained in the target region of property space after CRYSTAL17 band gap calculation. We confirmed that no final structure existed in the AFLOW database.26 This paper is available on arxiv under CC 4.0 license. (1) Sean M. Stafford, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA; (2) Alexander Aduenko, Moscow Institute of Physics and Technology, Moscow, Russia; (3) Marcus Djokic, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA; (4) Yu-Hsiu Lin, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA; (5) Jose L. Mendoza-Cortes, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA (Email: jmendoza@msu.edu). (1) Sean M. Stafford, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA; (2) Alexander Aduenko, Moscow Institute of Physics and Technology, Moscow, Russia; (3) Marcus Djokic, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA; (4) Yu-Hsiu Lin, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA; (5) Jose L. Mendoza-Cortes, Department of Chemical Engineering and Materials Science, Michigan State University, East Lansing, MI, 48824, USA (Email: jmendoza@msu.edu). Table of Links Abstract and Introduction Abstract and Introduction SALSA- (S)ubstitution, (A)pproximation, Evo(L)utionary (S)earch, and (A)B-Initio Calculations SALSA- (S)ubstitution, (A)pproximation, Evo(L)utionary (S)earch, and (A)B-Initio Calculations SALSA Applied to Photocatalytic Water-splitting SALSA Applied to Photocatalytic Water-splitting Discussion Discussion Methods Methods Conclusions, Data Availability Statement and References Conclusions, Data Availability Statement and References Appendix: Supplementary Material Appendix: Supplementary Material III. SALSA APPLIED TO PHOTOCATALYTIC WATER-SPLITTING We found that millions of candidate compounds could be generated from our initial dataset with the ion exchanges suggested by our substitution matrix. Of these, about 13,600 were compatible with our structural interpolation scheme, that is, they could be constructed as hybrids of compounds within our initial dataset of known semiconductors. See Section V B for details on this dataset construction. A. Candidate Compounds Overall, we found about 1250 hybrid compounds within our target region, including 484 within our ideal region. This corresponds to roughly one out of every 10 and 30 of all possible hybrids, respectively. Most interpolation pairings involved binary compounds with no elements in common so more hybrids were quaternary rather than ternary. Furthermore, the binary parents of ternary compounds tended to be located more closely to each other in property space, without any portion of the target region between them, so ternary compounds were relatively underrepresented in the regions of interest. The quaternary:ternary ratio was about 5:1 overall, 7:1 in the target region, and 8:1 in the ideal region. Figure 3 provides insight into how certain interpolation patterns emerged as dominant. These patterns can be understood in relation to the initial distribution of compounds in property space. Few initial compounds had acceptable Eg or φox and none had both simultaneously; however, acceptable φred was much more common. This combination advantageously positioned those with relatively high φox, especially the circled cluster containing the five highest φox compounds, because many partners were located across the ideal region from them. In fact, compounds from this cluster constituted one partner in nearly all interpolation pairings depicted in Figure 3, with the other partner being out-of-cluster and usually low φred. These pairings had the largest interpolation distance within the ideal region when the out-of-cluster partner was among the highest φox of the low-Eg compounds. Larger interpolation distance correlates with a greater number of possible hybrid compounds so this was the most dominant type of interpolation in our hybrid compound dataset. Thus we can roughly understand the interpolation opportunities available to our dataset by focusing on just a small subset of low-Eg and high-Eg compounds which are least oxidizable. The four highest φox compounds in the high-Eg cluster were AgBr, TiO2, AgCl, and CuCl, ordered by the number of hybrids derived from them. 95% of hybrid compounds had a parent in this group, including 42% from AgBr alone. The four highest φox compounds with low-Eg were the binary combinations of Pb and Ag with Se and Te. 40% of hybrids had a parent in this group and 36% that had one parent from each group. Table I provides some example hybrid compounds from the target region including three hybrids of different composition from pairs of AgBr and TiO2 with lower Eg compounds. The variety of hybrids included represents how different parents produced hybrids in different regions, e.g. TiO2 – PbO hybrids tended to have low Eg. B. Candidate Structures We used the procedure for USPEX and VASP laid out in Section V E to search for the crystal structures of hybrids in our target region. USPEX was able to converge structures for about 50 hybrid compounds. The elemental composition of these structures mostly coincides with the composition in the hybrid compounds highlighted in the previous Section. For example, Ag has the greatest occurrence by far, due to its presence in both the low and high Eg groups. However, Br has a surprisingly much lower occurrence and Cd has a relatively higher occurrence. Figure 4 and Table II show example results of USPEX converged structures. Figure 4 also connects shifts in composition to changes in structure and property space. USPEX allowed us to find hybrid structures that had different space groups from their parent structures. For example, PbCuSeCl has a space group of 156, which is different than any known PbSe or CuCl structure according to AFLOW.26 The approach of Hautier et al. 19 and most other materials searches using the substitution matrix technique would not have allowed the discovery of this structure because they enforced a restriction new candidate compounds had the same crystal structure as compounds from which they were derived. C. Final Water-Splitters Finally, we used the hybrid DFT code, CRYSTAL17, to conduct higher fidelity geometry optimization on our candidate structures. Many of the USPEX-converged structures had to be eliminated because the band gap derived from CRYSTAL17 moved them out of the target region of property space and others had to be eliminated because they would not converge. Structures containing Ag had larger discrepancies between their interpolated Eg and their CRYSTAL17-calculated Eg. On average the difference was 0.92 eV compared to 0.51 eV for non-Ag compounds. The largest difference was for Ag4Cl2Se which decreased by 1.17 eV, placing it slightly out of the ideal region. The surviving structures mostly had low symmetry. Figure 5 and Table III provide examples of final structures that remained in the target region of property space after CRYSTAL17 band gap calculation. We confirmed that no final structure existed in the AFLOW database.26 This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

SALSA Method Applied to Photocatalytic Water-splitting

Transforming Materials Discovery for Artificial Photosynthesis

Introduction: Large enterprises, which have more than one business unit, usually have more than one data platform environment. In fact, companies trying to be agile to make quick decisions, all have business unit-specific data platforms. With the computational advancements and rapid growth in data analytics and science, data integration tasks for data platforms are increasing exponentially as they need to support transactional as well as batch processing jobs. Here, the challenge is choosing the orchestration tool, as data integrations have to work in conjunction with different platforms. Though there are several tools in the market, in the article, we will cover how Apache airflow became the most trusted tool for the orchestration industry and its advantages. ETL: Traditional ETL (Extract, Transform, and loading) tools like Informatica, Ab Initio, and Talend all come with their own orchestration process. But as they are proprietary tools, it's expensive as well. Cloud service providers Azure, AWS (Amazon Web Services), and Google Cloud are all trying cash in the orchestration space so Azure has Synapse and AWS has Glue, Step function, etc. Basics: Orchestration is a simple or complex series of workflows defining a process. Workflows depend on one another to perform a business function or a series of business func‐tions calling other services. Then what is the ? process A process is a business function or set of business functions achieved by executing one or more tasks. Then what is the ? task A task is a step in a process. A task may call a service to perform a particular data-related function. Then what is the ? Service A service is software like Azure Synapse Analytics, Python data frame, or AWS DynamoDB that performs tasks. Common Problems in the Legacy Tools: The most common problem in the orchestration tools is their ability to successfully implement and maintain processes across the organization. Engineers and managers want to choose a tool that addresses the below issues: Dependency management: Lack of dependency management results in maintaining process-to-process and task-to-tasks dependency checks within the codebase along with business logic. This increases development time based on the number of processes or tasks involved in a workflow. Task modification to existing processes: Workflow is defined as each step within the code. Embedding the workflow in the code requires manual effort to fully review the code to make changes to one or more tasks resulting in additional time involved in project enhancements and testing. Troubleshooting: Troubleshooting a particular process should not require the need to understand the dependencies, as it results in additional support time. Restart: It should be fairly easy to restart the failed task or trigger an ad-hoc task. Orchestration should not warrant the need to rerun the entire process if a particular task within the process failed. User-friendly interface to track the progress: Orchestration tools should provide best in class user interface with the details to visualize the tasks involved within the processes in one place. Engineers investigating the issue should have a UI to understand the flow of a particular process. UI should display the current process in execution, the time it's taking, etc. Apache Airflow Airflow is a platform to programmatically author, schedule, and monitor workflows geared at data pipelining. Airflow is used to author workflows as DAGs of tasks. The Airflow scheduler executes tasks on an array of workers while following the specified dependencies. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed. Code Development: Authoring a data pipeline using existing tasks involves writing a Python module and deploying it to the Airflow installation. Adding a new kind of task to an airflow installation involves writing and deploying Python code in packages called plugins. This aligns with the primary coding skill set of Data Engineers and scientists, which is Python. Benefits At a high level, process dependency management is easy to develop and maintain via user interface (UI) and support in Airflow, which helps immensely to save time for developers and supporting resources. DAG creation is all in Python scripting language which is the primary coding skillset for Data Engineers. Changes within Airflow can be captured using the GIT repository enabling version controls and change management practices. Airflow has good documentation and an active internal community for answering questions. What are the skillsets required to develop and maintain Airflow? Python is a major skill required to use Airflow. Initial learnings are required to set up Airflow and understand the architecture so it's evident for future maintenance. Can Engineers visualize the Jobs? Yes, Airflow has a Web interface to start and stop processes and also to monitor the current status of the processes. Does Airflow support any integration with Data Quality checkers? DataDog can be integrated with Airflow. How do we make ensure Airflow’s availability? Airflow workers and Web servers can be load balanced. Airflow Scheduler is a single point of failure. How does Airflow scale? Airflow Web server can be scaled based on the number of requests. Airflow workers can be scaled up and down based on requirements.

Stories, News & Mentions about INIZIO

Inizio’s Stories on HackerNoon