As the first step, SEL will set up a Board of Directors once the Monitoring Committee appointed by the NCLT gets dissolved. SEL will kick start operations as soon as the takeover process is completed.

Swan Energy to take over Reliance Naval and Engineering, shares jump 6%

The Marine Engineering MSc is concerned with the design, analysis and operation of machinery and systems for merchant and naval ships and submarines. The programme covers a wide range of engineering subjects relevant to the development and procurement of ...

Marine Engineering MSc

"As China continues to expand domestic consumption, accelerate industrial upgrading and promote high-level opening-up, I believe that China has ample policy space to counter economic downturn pressures and explore new growth opportunities in sectors such as new energy, artificial intelligence and green low-carbon industries," Chen added.

China retains strong appeal for investment

This 10-week summer program places science, engineering, and mathematics students at media organizations nationwide. Fellows use their academic training as they research, write, and report today’s headlines, sharpening their abilities to communicate ...

AAAS Mass Media Science & Engineering Fellowship

Genetic engineering is the act of modifying the genetic makeup of an organism. Modifications can be generated by methods such as gene targeting, nuclear transplantation, transfection of synthetic ...

Genetic engineering articles from across Nature Portfolio

Suzlon stock went up to 5% in a single day and the overall sentiment of the stock is looking bullish. It took support above a trendline and for the past few days, it has been hitting the upper circuit.

Breakout Stocks: How IRB Infrastructure, Triveni Engineering and Suzlon are looking on charts for Thursday’s trade

BMW has been down the small, electrified city car route before with the i3, but this new attempt at capturing the compact car segment won't make the same mistakes. While the i3 was immensely ...

BMW i3 To Get Neue Klasse Successor

The streamlining plan would merge its engineering technology and safety groups, Rolls-Royce said, and as part of that its chief technology officer Grazia Vittadini would leave the business in April 2024.

Rolls-Royce to cut up to 2,500 roles in latest efficiency drive

The rigorous engineering education we provide to students at all stages of their lives is the center of who we are as an institution. A Duke Engineering education leads to lives of purpose and integrity that are as rewarding as they are impactful.

Our Story

i3 engineering

i3engineering

I’ve recently discovered an emerging area of inquiry called DesignOps. The definition of DesignOps is still very much being worked out, which is not surprising given its newness. There seems to be some agreement, though, that DesignOps takes inspiration from DevOps. This inspiration is very pleasing. It’s important, though, for designers to understand what DevOps is, and more importantly, what it isn’t. Unfortunately, I’m seeing some misunderstanding in this area, which I believe warrants clarification. First of all, DevOps doesn’t stand for “developer operations”, or for that matter, for anything else aside from itself. Its name reflects the view of development and operations as intimately linked, collaborating, and mutually influencing parts of a larger whole. That larger whole is, quite simply, “DevOps”. DevOps applies systems thinking to IT by treating the relationship between development and operations as being just as important as, if not more so than, the individual roles and activities. DevOps also is not an attempt to “remove QA and Sysadmin from the operations equation”. The point is to get Dev and QA and Sysadmin working together and influencing each other. Different organizations of different sizes may have different mixes of people with particular roles and or titles. Especially in startups, it may be the case that people with one title do all three jobs. It is often the case that automation reduces the number of people of a given role you need to do a given amount of work. But getting rid of people or roles is not the point. DevOps’s core insight is that reducing the distance between people, teams, and activities, combined with reducing the batch size of your work, allows you to deliver more value, more continuously, with greater quality. Furthermore, you do so, not just to go faster, but to make the feedback loop between the customer and the company more intimate. Or, to express it in the language of the Three Ways of DevOps, Flow + Feedback enable Continuous Learning. Enabling adaption is thus DevOps’ end goal. It is my hope that DevOps will provide DesignOps with inspiration on this level. What I really hope is that a conversation will happen between them about how to apply systems thinking to the entire digital service delivery lifecycle. As Kate Ivey-Williams said in , “a service is an ever-evolving, iterative thing”. For that model to work, we need to unify our understanding and practices from design all the way through to operations (and back again). In fact, when I first saw the word “DesignOps”, I thought it referred to getting designers and IT operations folks understanding and working with each other. her wonderful post about service design Those of you who follow my work will recognize a broken record in my use of the phrase “unifying design and operations”. Nonetheless, I see it as a critical next step in our efforts to create adaptive, continuously learning organizations. I want to encourage everyone, whether designers, product managers, or DevOps practitioners, to help create a conversation that both pursues and expresses this vision. If my call to action strikes a chord, I’d love to hear from you at or on Twitter at @jeffsussna. https://ingineering.it

DevOps for Designers

The art of prompt engineering is becoming a critical skill for effectively harnessing AI's potential as artificial intelligence continues to evolve. Yet, for all its importance, prompt engineering is often reduced to simplistic advice—tips like “be specific” or quick hacks such as “let’s think step by step.” While helpful for beginners, these pointers barely scratch the surface. True prompt engineering requires a much deeper understanding of the mechanics behind AI responses, moving far beyond what’s often little more than zero-shot Google searching. Here we go through some foundational principles to explain why good prompting actually works. Embeddings: The Foundation of Prompt Understanding One of the foundational concepts in effective prompting is understanding embeddings. When we input a prompt into a model, it doesn’t process language as we do; instead, each word or phrase is transformed into an embedding, a numerical representation that captures both the meaning and context of the language. These embeddings act as the model’s internal mapping, helping it interpret relationships, nuances, and associations between words. For prompt engineers, understanding embeddings is essential for prompt intuition. The way a word is represented in embedding space determines how the model interprets it, affecting the content and focus of responses. For example, if a prompt requests information on “manufacturing,” the model will pull content based on embeddings linked to those terms, potentially drawing from areas like production processes, supply chain logistics, or technological advancements. A prompt engineer can influence these responses by skillfully crafting the wording, improving the relevance and depth of the output. Understanding embeddings also empowers prompt engineers to chain prompts better, where each response builds on previous interactions. By crafting sequences with embedding relationships in mind, prompt engineers create complex, context-aware conversations that adapt to specific needs, producing detailed, responsive outputs. The Role of Attention: Guiding Model Focus Another core component in prompt engineering is attention. Unlike human attention, which can dynamically shift between different topics, AI attention assigns weight to relationships between words within the prompt and response text. Attention helps the model prioritize certain parts of the input, and a well-crafted prompt uses this to shape the model’s focus. Take, for instance, a prompt asking for a nuanced view of a cat sitting on a mat. Prompt engineers can guide the model to emphasize relevant aspects of the prompt's subject, such as the cat's posture, the texture and color of the mat, or the overall scene’s ambiance, leading to a more detailed and vivid response. This approach to prompting. known as directional stimulus, allows the extraction of richer, more relevant answers from the model. RAG: Retrieval-Augmented Generation for Enhanced Responses Retrieval-Augmented Generation (RAG) combines generative AI with retrieval systems to improve the accuracy and relevance of responses. Traditional prompt engineering relies on a model’s internal knowledge, which can be outdated or incomplete. RAG addresses this by adding a retrieval layer that pulls in current, contextually relevant information from external sources like databases or search engines, enabling the model to provide more accurate, up-to-date answers. RAG is transformative for prompt engineering. It allows engineers to structure prompts that direct both the retrieval and generative components to work together, especially valuable for complex or fast-evolving subjects. For instance, a prompt about “recent changes in data privacy laws” could trigger the model to fetch the latest legal documents or news, which it then combines with internal information via RAG to synthesize into a coherent response. This dual capability reduces hallucinations and keeps the model grounded in real-time, reliable information. Prompt engineers working within RAG frameworks can therefore create interactions that are both innovative and anchored in current knowledge, adding an extra layer of sophistication to the AI’s output. Image Prompting and the Diffusion Process In image generation, prompt engineering takes on another layer of complexity. Here, prompts are not just about words but about guiding a model through the diffusion process—a technique where random noise is iteratively transformed into a coherent image. The diffusion process relies on iterative adjustments, and each cycle builds upon the previous one to refine an image based on the prompt's guidance. When crafting prompts for image generation, understanding diffusion is essential. Descriptions like “award winning photography” or “fashion photography” can yield varying results depending on how the model interprets "award winning" or “fashion.” Prompt engineers must understand the model’s interpretation of spatial relationships, color, and composition to produce a desired outcome. It’s about more than aesthetics; it’s about leading the model through each step to align with the intended visual concept. Hallucinations and the Pitfalls of Poor Prompting One of the most challenging aspects of AI prompting is managing hallucinations—when a model produces content that sounds plausible but is entirely fabricated. Hallucinations often result from vague or poorly structured prompts, which leave too much to the model’s interpretation. In fields like healthcare or finance, where accuracy is critical, even a small hallucination can have serious consequences. Skilled prompt engineers prevent this by constructing prompts that minimize ambiguity, verifying outputs with fact-checking prompts, and maintaining strict clarity and focus. Why Prompt Engineering is a Real Skill Ultimately, prompt engineering is a genuine skill that combines language intuition with a deep technical understanding of AI mechanics. Unlike simple search queries, prompt engineers with knowledge of embeddings, attention mechanisms, and model limitations are able to craft better prompts. Mastering these elements enables prompt engineers to shape responses that are precise, reliable, and often surprisingly creative. The need for skilled prompt engineers will only grow as AI technology continues to integrate into industries from education to entertainment. They’ll be essential not just for generating content, but for refining AI applications, troubleshooting outputs, and optimizing model performance across fields. Far from a passing trend, prompt engineering is shaping up to be one of the defining skills of the AI era—an essential tool for building communication between humans and machines. About Me: 25+ year IT veteran combining data, AI, risk management, strategy, and education. 4x global hackathon winner and social impact from data advocate. Currently working to jumpstart the AI workforce in the Philippines. Learn more about me here. The art of prompt engineering is becoming a critical skill for effectively harnessing AI's potential as artificial intelligence continues to evolve. Yet, for all its importance, prompt engineering is often reduced to simplistic advice—tips like “be specific” or quick hacks such as “let’s think step by step.” While helpful for beginners, these pointers barely scratch the surface. True prompt engineering requires a much deeper understanding of the mechanics behind AI responses, moving far beyond what’s often little more than zero-shot Google searching. Here we go through some foundational principles to explain why good prompting actually works. Embeddings: The Foundation of Prompt Understanding One of the foundational concepts in effective prompting is understanding embeddings. When we input a prompt into a model, it doesn’t process language as we do; instead, each word or phrase is transformed into an embedding, a numerical representation that captures both the meaning and context of the language. These embeddings act as the model’s internal mapping, helping it interpret relationships, nuances, and associations between words. a numerical representation For prompt engineers, understanding embeddings is essential for prompt intuition. The way a word is represented in embedding space determines how the model interprets it, affecting the content and focus of responses. For example, if a prompt requests information on “manufacturing,” the model will pull content based on embeddings linked to those terms, potentially drawing from areas like production processes, supply chain logistics, or technological advancements. A prompt engineer can influence these responses by skillfully crafting the wording, improving the relevance and depth of the output. Understanding embeddings also empowers prompt engineers to chain prompts better , where each response builds on previous interactions. By crafting sequences with embedding relationships in mind, prompt engineers create complex, context-aware conversations that adapt to specific needs, producing detailed, responsive outputs. chain prompts better The Role of Attention: Guiding Model Focus Another core component in prompt engineering is attention. Unlike human attention, which can dynamically shift between different topics, AI attention assigns weight to relationships between words within the prompt and response text. Attention helps the model prioritize certain parts of the input, and a well-crafted prompt uses this to shape the model’s focus. Take, for instance, a prompt asking for a nuanced view of a cat sitting on a mat. relationships between words Prompt engineers can guide the model to emphasize relevant aspects of the prompt's subject, such as the cat's posture, the texture and color of the mat, or the overall scene’s ambiance, leading to a more detailed and vivid response. This approach to prompting. known as directional stimulus , allows the extraction of richer, more relevant answers from the model. directional stimulus RAG: Retrieval-Augmented Generation for Enhanced Responses Retrieval-Augmented Generation (RAG) combines generative AI with retrieval systems to improve the accuracy and relevance of responses. Traditional prompt engineering relies on a model’s internal knowledge, which can be outdated or incomplete. RAG addresses this by adding a retrieval layer that pulls in current, contextually relevant information from external sources like databases or search engines, enabling the model to provide more accurate, up-to-date answers. adding a retrieval layer RAG is transformative for prompt engineering. It allows engineers to structure prompts that direct both the retrieval and generative components to work together, especially valuable for complex or fast-evolving subjects. For instance, a prompt about “recent changes in data privacy laws” could trigger the model to fetch the latest legal documents or news, which it then combines with internal information via RAG to synthesize into a coherent response. This dual capability reduces hallucinations and keeps the model grounded in real-time, reliable information. Prompt engineers working within RAG frameworks can therefore create interactions that are both innovative and anchored in current knowledge, adding an extra layer of sophistication to the AI’s output. Image Prompting and the Diffusion Process In image generation, prompt engineering takes on another layer of complexity. Here, prompts are not just about words but about guiding a model through the diffusion process—a technique where random noise is iteratively transformed into a coherent image. The diffusion process relies on iterative adjustments, and each cycle builds upon the previous one to refine an image based on the prompt's guidance. random noise is iteratively transformed When crafting prompts for image generation, understanding diffusion is essential. Descriptions like “award winning photography” or “fashion photography” can yield varying results depending on how the model interprets "award winning" or “fashion.” Prompt engineers must understand the model’s interpretation of spatial relationships, color, and composition to produce a desired outcome. It’s about more than aesthetics; it’s about leading the model through each step to align with the intended visual concept. Hallucinations and the Pitfalls of Poor Prompting One of the most challenging aspects of AI prompting is managing hallucinations—when a model produces content that sounds plausible but is entirely fabricated. Hallucinations often result from vague or poorly structured prompts , which leave too much to the model’s interpretation. In fields like healthcare or finance, where accuracy is critical, even a small hallucination can have serious consequences. vague or poorly structured prompts Skilled prompt engineers prevent this by constructing prompts that minimize ambiguity , verifying outputs with fact-checking prompts, and maintaining strict clarity and focus. prompts that minimize ambiguity Why Prompt Engineering is a Real Skill Ultimately, prompt engineering is a genuine skill that combines language intuition with a deep technical understanding of AI mechanics. Unlike simple search queries, prompt engineers with knowledge of embeddings, attention mechanisms, and model limitations are able to craft better prompts. Mastering these elements enables prompt engineers to shape responses that are precise, reliable, and often surprisingly creative. The need for skilled prompt engineers will only grow as AI technology continues to integrate into industries from education to entertainment. They’ll be essential not just for generating content, but for refining AI applications, troubleshooting outputs, and optimizing model performance across fields. Far from a passing trend, prompt engineering is shaping up to be one of the defining skills of the AI era—an essential tool for building communication between humans and machines. About Me: 25+ year IT veteran combining data, AI, risk management, strategy, and education. 4x global hackathon winner and social impact from data advocate. Currently working to jumpstart the AI workforce in the Philippines. Learn more about me here . here

Explaining Prompt Engineering

প্রম্পট ইঞ্জিনিয়ারিং ব্যাখ্যা করা

Erklärung zu Prompt Engineering

Explicando la ingeniería rápida

Explication de l'ingénierie rapide

प्रॉम्प्ट इंजीनियरिंग की व्याख्या

プロンプトエンジニアリングの説明

프롬프트 엔지니어링 설명

Explicando a Engenharia Prompt

Объяснение оперативной инженерии

İstemli Mühendisliğin Açıklanması

Giải thích về Kỹ thuật Prompt

解释快速工程

Authors:
(1) Joseph Latessa, Department of Computer Science Wayne State University, Detroit MI USA (jlatessa@wayne.edu);
(2) Aadi Huria, Senior, Salem High School Canton, MI USA (huria.aadi@gmail.com);
(3) Deepak Raju, Senior, Salem High School, Canton MI USA (Deepak.Raju294@outlook.com). Table of Links Abstract and Introduction Related Work Project Prerequisites Project Implementation Insight and Reflections Conclusions, Acknowledgement and References 6 CONCLUSION For motivated independent study students, learning software engineering concepts such as version control and continuous integration will benefit them in their future studies. As demonstrated here, this content is accessible to students even at an early point in their computer science education. A special honors project or a supplemental one-credit-hour tools and techniques course taken as a corequisite to the traditional introduction to programming track could be a next step to further develop, scale, and generalize the material and unique format devised and embarked upon for this project. ACKNOWLEDGMENTS We would like to acknowledge Loren Schwiebert at Wayne State University for his assistance, feedback, and advice throughout the development and implementation of this project. We would also like to acknowledge the JETSCAPE Collaboration [1] and GOMC [2] for providing the source repositories and applications on which our automated tests were developed. We also acknowledge LinkChecker [12], GitHub Actions HTML5 Validator [10], and svenkreiss/html5validator [11] as the marketplace and opensource utilities discussed and applied as part of this project. REFERENCES [1] JETSCAPE Website: https://jetscape.org. [2] GOMC - GPU-Optimized Monte Carlo Website: https://gomc-wsu.org. [3] Chib, Siddhartha, and Edward Greenberg. “Understanding the MetropolisHastings Algorithm.” The American Statistician, vol. 49, no. 4, 1995, pp. 327– 35. JSTOR, https://doi.org/10.2307/2684568. Accessed 17 Aug. 2023. [4] Richard Glassey. 2019. Adopting Git/Github within Teaching: A Survey of Tool Support. In Proceedings of the ACM Conference on Global Computing Education (CompEd '19). Association for Computing Machinery, New York, NY, USA, 143–149. https://doi.org/10.1145/3300115.3309518 [5] Yu-Cheng Tu, Valerio Terragni, Ewan Tempero, Asma Shakil, Andrew Meads, Nasser Giacaman, Allan Fowler, and Kelly Blincoe. 2022. GitHub in the Classroom: Lessons Learnt. In Proceedings of the 24th Australasian Computing Education Conference (ACE '22). Association for Computing Machinery, New York, NY, USA, 163–172. https://doi.org/10.1145/3511861.3511879 [6] Lassi Haaranen and Teemu Lehtinen. 2015. Teaching Git on the Side: Version Control System as a Course Platform. In Proceedings of the 2015 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '15). Association for Computing Machinery, New York, NY, USA, 87–92. https://doi.org/10.1145/2729094.2742608 [7] David S. Janzen and Hossein Saiedian. 2006. Test-driven learning: intrinsic integration of testing into the CS/SE curriculum. SIGCSE Bull. 38, 1 (March 2006), 254–258. https://doi.org/10.1145/1124706.1121419 [8] B. P. Eddy et al., "A Pilot Study on Introducing Continuous Integration and Delivery into Undergraduate Software Engineering Courses," 2017 IEEE 30th Conference on Software Engineering Education and Training (CSEE&T), Savannah, GA, USA, 2017, pp. 47-56, doi: 10.1109/CSEET.2017.18. [9] Alex Radermacher and Gursimran Walia. 2013. Gaps between industry expectations and the abilities of graduates. In Proceeding of the 44th ACM technical symposium on Computer science education (SIGCSE '13). Association for Computing Machinery, New York, NY, USA, 525–530. https://doi.org/10.1145/2445196.2445351 [10] HTML5 Validator - GitHub Marketplace: https://github.com/marketplace/actions/html5-validator. [11] GitHub - svenkreiss/html5validator: Command line tool to validate HTML5 files. Great for continuous integration.: 2022. https://github.com/svenkreiss/html5validator. [12] LinkChecker: 2022. https://pypi.org/project/LinkChecker/. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Joseph Latessa, Department of Computer Science Wayne State University, Detroit MI USA (jlatessa@wayne.edu); (2) Aadi Huria, Senior, Salem High School Canton, MI USA (huria.aadi@gmail.com); (3) Deepak Raju, Senior, Salem High School, Canton MI USA (Deepak.Raju294@outlook.com). Authors: Authors: (1) Joseph Latessa, Department of Computer Science Wayne State University, Detroit MI USA (jlatessa@wayne.edu); (2) Aadi Huria, Senior, Salem High School Canton, MI USA (huria.aadi@gmail.com); (3) Deepak Raju, Senior, Salem High School, Canton MI USA (Deepak.Raju294@outlook.com). Table of Links Abstract and Introduction Abstract and Introduction Related Work Related Work Project Prerequisites Project Prerequisites Project Implementation Project Implementation Insight and Reflections Insight and Reflections Conclusions, Acknowledgement and References Conclusions, Acknowledgement and References 6 CONCLUSION For motivated independent study students, learning software engineering concepts such as version control and continuous integration will benefit them in their future studies. As demonstrated here, this content is accessible to students even at an early point in their computer science education. A special honors project or a supplemental one-credit-hour tools and techniques course taken as a corequisite to the traditional introduction to programming track could be a next step to further develop, scale, and generalize the material and unique format devised and embarked upon for this project. ACKNOWLEDGMENTS We would like to acknowledge Loren Schwiebert at Wayne State University for his assistance, feedback, and advice throughout the development and implementation of this project. We would also like to acknowledge the JETSCAPE Collaboration [1] and GOMC [2] for providing the source repositories and applications on which our automated tests were developed. We also acknowledge LinkChecker [12], GitHub Actions HTML5 Validator [10], and svenkreiss/html5validator [11] as the marketplace and opensource utilities discussed and applied as part of this project. REFERENCES [1] JETSCAPE Website: https://jetscape.org. [2] GOMC - GPU-Optimized Monte Carlo Website: https://gomc-wsu.org. [3] Chib, Siddhartha, and Edward Greenberg. “Understanding the MetropolisHastings Algorithm.” The American Statistician, vol. 49, no. 4, 1995, pp. 327– 35. JSTOR, https://doi.org/10.2307/2684568. Accessed 17 Aug. 2023. [4] Richard Glassey. 2019. Adopting Git/Github within Teaching: A Survey of Tool Support. In Proceedings of the ACM Conference on Global Computing Education (CompEd '19). Association for Computing Machinery, New York, NY, USA, 143–149. https://doi.org/10.1145/3300115.3309518 [5] Yu-Cheng Tu, Valerio Terragni, Ewan Tempero, Asma Shakil, Andrew Meads, Nasser Giacaman, Allan Fowler, and Kelly Blincoe. 2022. GitHub in the Classroom: Lessons Learnt. In Proceedings of the 24th Australasian Computing Education Conference (ACE '22). Association for Computing Machinery, New York, NY, USA, 163–172. https://doi.org/10.1145/3511861.3511879 [6] Lassi Haaranen and Teemu Lehtinen. 2015. Teaching Git on the Side: Version Control System as a Course Platform. In Proceedings of the 2015 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '15). Association for Computing Machinery, New York, NY, USA, 87–92. https://doi.org/10.1145/2729094.2742608 [7] David S. Janzen and Hossein Saiedian. 2006. Test-driven learning: intrinsic integration of testing into the CS/SE curriculum. SIGCSE Bull. 38, 1 (March 2006), 254–258. https://doi.org/10.1145/1124706.1121419 [8] B. P. Eddy et al., "A Pilot Study on Introducing Continuous Integration and Delivery into Undergraduate Software Engineering Courses," 2017 IEEE 30th Conference on Software Engineering Education and Training (CSEE&T), Savannah, GA, USA, 2017, pp. 47-56, doi: 10.1109/CSEET.2017.18. [9] Alex Radermacher and Gursimran Walia. 2013. Gaps between industry expectations and the abilities of graduates. In Proceeding of the 44th ACM technical symposium on Computer science education (SIGCSE '13). Association for Computing Machinery, New York, NY, USA, 525–530. https://doi.org/10.1145/2445196.2445351 [10] HTML5 Validator - GitHub Marketplace: https://github.com/marketplace/actions/html5-validator. [11] GitHub - svenkreiss/html5validator: Command line tool to validate HTML5 files. Great for continuous integration.: 2022. https://github.com/svenkreiss/html5validator. [12] LinkChecker: 2022. https://pypi.org/project/LinkChecker/. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Building the Foundation: The Benefits of Early Software Engineering Education

Authors: (1) LI LI, Beihang University, China; (2) XIANG GAO, Beihang University, China; (3) HAILONG SUN, Beihang University, China; (4) CHUNMING HU, Beihang University, China; (5) XIAOYU SUN, The Australian National University, Australia; (6) HAOYU WANG, Huazhong University of Science and Technology, China; (7) HAIPENG CAI, Washington State University, Pullman, USA; (8) TING SU, East China Normal University, China; (9) XIAPU LUO, The Hong Kong Polytechnic University, China; (10) TEGAWENDÉ F. BISSYANDÉ, University of Luxembourg, Luxembourg; (11) JACQUES KLEIN, University of Luxembourg, Luxembourg; (12) JOHN GRUNDY, Monash University, Australia; (13) TAO XIE, Peking University, China; (14) HAIBO CHEN, Shanghai Jiao Tong University, China; (15) HUAIMIN WANG, National University of Defense Technology, China. Table of Links Introduction Background of OpenHarmony The State Of OpenHarmony Ecosystem Overview Of Mobile Software Engineering The Research Roadmap Discussion Related Work Conclusion & References 7 RELATED WORK OpenHarmony software engineering is in its early stage and there are only limited works contributed to this field. Indeed, as highlighted in Section 3.4, there are only 8 papers presented on this aspect. In this section, we will not discuss these OpenHarmony-related works anymore. Instead, we take this opportunity to highlight related works that provide a research roadmap or position statement for guiding a new research field, or a survey including literature reviews for summarizing a mature research direction. We now highlight the representative ones. . One of the most representative research roadmap reports is the one presented by Cheng et al. [22] who have proposed to conduct software engineering research for self-adaptive systems. After thorough discussions among the authors at a Dagstuhl seminar on Software Engineering for Self-Adaptive Systems, the authors have identified four views that are deemed essential to the software engineering of self-adaptive systems. For each view, the authors then summarize the state-of-the-art and highlight the challenges that should be addressed in order to achieve the final goal, i.e., the software is able to automatically cope with the complexity of today’s software-intensive systems. The authors have released another version (called the second research roadmap) five years later after the success of the first version. The goal of this second roadmap paper [25] remains the same, i.e., to summarize the state-of-the-art and to identify critical challenges for the systematic software engineering of self-adaptive systems. Other representative research roadmap papers include the one proposed by France et al. [34] who advocate model-driven development of complex software as well as the one proposed by Papazoglou et al. [85] who advocate service-oriented computing as a new computing paradigm for supporting the development of rapid, low-cost and easy composition of distributed applications. Both of these works have summarized the state-of-the-art and challenges faced by ongoing research activities. More recently, McDermott et al. [79] present a research roadmap about Artificial Intelligence for Software Engineering (AI4SE) and Software Engineering for Artificial Intelligence (SE4AI), presenting key aspects aiming at enabling traditional systems engineering practice automation (AI4SE), and encourage new systems engineering practices supporting a new wave of automated, adaptive, and learning systems (SE4AI). Research Roadmap A literature review involves surveying scholarly sources (mainly research publications) on a specific topic, aiming to provide an overview of the state-of-the-art that is further backed up with a critical evaluation of the material. Except for providing a reflection on the past, it also gives a clear picture of the state of knowledge on the subject that is helpful for guiding future research directions. Because of the aforementioned benefits, in this work, we have resorted to surveying the literature review papers (instead of the majority of primary publications) presented in the field of mobile software engineering. Actually, conducting a survey of surveys is not new to the community. Our fellow researchers have explored this type of study in various domains when the number of primary publications kept increasing until it became difficult to follow the growing body of literature papers in the field. For example, AI-Zewairi et al. [3] have conducted a survey of surveys related to agile software development methodologies, which have gained rigorous attention in the software engineering community with an excessive number of research studies published. As another example, McNabb et al. [80] have presented to the community a survey of surveys about information visualization, which has also become extremely popular and the number of publications has become increasingly difficult to follow. Other representative works include the one proposed by Giraldo et al. [41] who have proposed a survey of surveys on the topic of security and privacy in Cyber-physical systems as well as the one proposed by Chatzimparmpas et al. [20] who have conducted a survey of surveys on the use of visualization for interpreting machine learning models. Literature Review. This paper is under CC 4.0 license. available on arxiv

Software Engineering for OpenHarmony: Related Work

OpenHarmony-এর জন্য সফটওয়্যার ইঞ্জিনিয়ারিং: সম্পর্কিত কাজ

Softwareentwicklung für OpenHarmony: Verwandte Arbeiten

Ingeniería de software para OpenHarmony: trabajos relacionados

Génie logiciel pour OpenHarmony : travaux connexes

ओपनहार्मनी के लिए सॉफ्टवेयर इंजीनियरिंग: संबंधित कार्य

OpenHarmony のソフトウェア エンジニアリング: 関連作業

OpenHarmony를 위한 소프트웨어 엔지니어링: 관련 작업

Engenharia de software para OpenHarmony: trabalhos relacionados

Разработка программного обеспечения для OpenHarmony: сопутствующие работы

OpenHarmony için Yazılım Mühendisliği: İlgili Çalışmalar

Kỹ thuật phần mềm cho OpenHarmony: Công việc liên quan

OpenHarmony 软件工程：相关工作

Software Engineering for OpenHarmony—A Research Roadmap: Conclusion & References

Software Engineering for OpenHarmony: The Research Roadmap

Authors: (1) LI LI, Beihang University, China; (2) XIANG GAO, Beihang University, China; (3) HAILONG SUN, Beihang University, China; (4) CHUNMING HU, Beihang University, China; (5) XIAOYU SUN, The Australian National University, Australia; (6) HAOYU WANG, Huazhong University of Science and Technology, China; (7) HAIPENG CAI, Washington State University, Pullman, USA; (8) TING SU, East China Normal University, China; (9) XIAPU LUO, The Hong Kong Polytechnic University, China; (10) TEGAWENDÉ F. BISSYANDÉ, University of Luxembourg, Luxembourg; (11) JACQUES KLEIN, University of Luxembourg, Luxembourg; (12) JOHN GRUNDY, Monash University, Australia; (13) TAO XIE, Peking University, China; (14) HAIBO CHEN, Shanghai Jiao Tong University, China; (15) HUAIMIN WANG, National University of Defense Technology, China. Table of Links Introduction Background of OpenHarmony The State Of OpenHarmony Ecosystem Overview Of Mobile Software Engineering The Research Roadmap Discussion Related Work Conclusion & References 4 OVERVIEW OF MOBILE SOFTWARE ENGINEERING In this work, we are interested in building a research roadmap for conducting software engineering research for OpenHarmony. Unfortunately, since OpenHarmony is still in its early stages, there is not much work proposed for that. Nevertheless, we believe all the research efforts contributed to improving the Android and iOS ecosystem could be also conducted for OpenHarmony. Therefore, in this section, we first resort to a systematic literature review to understand the status quo of mobile software engineering research. We will then leverage the empirical observations to form our research roadmap dedicated to OpenHarmony. In this work, we conduct the systematic literature review following the methodologies outlined by Brereton et al. [16] and Li et al. [60]. Fig. 6 highlights the working process. In we plan Step 1, to investigate the latest research advancements in the area of mobile software engineering by answering the following research question. RQ: What problems are targeted by our fellow researchers in the MSE community and how they are resolved? Then, in , we identify the search keywords that could be used to find all the relevant publications, in order to answer the pre-defined research questions. When we started to do that, we immediately realized that the number of primary publications was too huge, it is literally impossible to manually read all of them. Indeed, taking Android-related research alone, there are already over 7,500 papers recorded on DBLP. If one is able to read a paper in 2 hours, it would still require years to complete. To mitigate this problem, we resort to only considering the existing survey and literature review papers, for which our fellow researchers have already systematically reviewed the different aspects of mobile software engineering. We believe these survey papers are representative of the status quo of mobile software engineering research. To this end, we identify the search keywords based on these concerns and the list of included keywords is summarized in Table 5. In total, we have identified two groups of keywords, represented as G1 and G2, respectively. We then form the query based on this rule[9] for which we require it to contain at least one keyword from each group. Step 2 After the query is formed, in , we directly applied it to search relevant studies in the Computer Science Bibliography (DBLP) Database. This step gives us 65 papers that are potentially relevant to our study. After that, in , we refine the gathered list of relevant papers by manually checking their relevance to mobile software engineering (i.e., could indeed be helpful for answering the aforementioned research question). Specifically, we filter out the less relevant papers based on the following set of exclusion criteria. Step 3 Step 4 (1) Since we only consider survey or literature review papers, all the non-survey papers are simply excluded from our study. (2) Although there are some papers that meet our selection criteria (i.e., whose title contains the group keywords in Table 5), their topics may not strictly fall into the software engineering category. We initially eliminate these papers by manually reviewing the abstracts and identifying those that have the potential to provide guidance for OpenHarmony. Once the irrelevant papers are filtered out, we then conduct a backward snowballing (i.e., ) by scanning all the referenced papers and checking if they should also be considered for our study. We remind the readers that we have cross-checked the results (i.e., ) in all the previous two steps (i.e., exclusion criteria filtering and backward snowballing) to ensure the reliability of the results. Step 5 Step 6 We were able to eventually collect 39 papers to answer our research question defined at the beginning of this study. Table 6 enumerates the list of selected papers, including their publication year and venue. Once the relevant papers are collected, we carefully read all of them and attempt to extract the relevant data (i.e., ) from each paper to answer the research question. Specifically, we aim to extract the following two types of information: (1) Targeted Problems, which involve understanding the issues within the Android/iOS ecosystem that have been identified by our MSE researchers as problems needing resolution to create a more user-friendly mobile ecosystem, and (2) Fundamental Techniques, aimed at discovering the techniques required to address the various challenges in the mobile community. Considering that OpenHarmony may encounter similar issues to those faced by Android and iOS, we argue that insights gained from exploring these two aspects could prove valuable in shaping the roadmap for conducting software engineering research for OpenHarmony. Furthermore, similar to our approach in identifying relevant papers, we have conducted cross-checks of our observations, involving at least two authors, to ensure the reliability of these observations, thereby enhancing the trustworthiness of the research roadmap. Step 7 4.1 Problem Before going into the details in summarizing the top problems targeted by our fellow researchers in MSE, we first present the major participants (or artifacts) involved in MSE research. These participants have been closely associated with the top problems identified and handled in MSE. As illustrated in Fig. 7, play a core role in MSE, who contribute to the ecosystem by implementing based on the (also known as the SDK) provided by Google, along with various third-party libraries that are pre-developed for facilitating app developments. The libraries also include the ones used to provide , which also play a crucial role in Android as they are the major source for app developers to make profits.[10] When there are problems encountered while developing an app, developers frequently resort to te (such as Stack Overflow) to search for solutions. The app’s source code is often managed on code hosting websites such as Github, which is also one of the most important resources leveraged by mining software repository researchers to learn for improving Android developers mobile apps Android framework advertisements question and answer websi apps. Once the apps are developed, they will be uploaded to such as the official Google Play store, on which various associated with the app (such as app’s description, name, authors, etc.) will also be provided. The app stores are the main portal for users to find and install apps. Except for searching and installing apps, app stores also provide a platform for users to leave feedback (i.e., , which could be complaints about defects or suggestions regarding new app features) for their apps on dedicated pages. app stores metadata user comments We now highlight the top problems targeted by our fellow researchers (cf. Table 7). These top problems could be applied to any of the aforementioned participants highlighted in Fig. 7. The problems are mainly grouped into nine categories, including app development, app deployment, user experience, security and privacy, quality, reliability, performance, energy, and socio-technical issues. To help readers better understand each of the categories (i.e., the actual problems handled by our fellow researchers), we also provide various problem examples in the second column of the table. 4.2 Technique To solve the above software engineering problems, researchers have proposed various kinds of techniques. Note that, while there are more techniques designed to solve the above problems, e.g., trust environment execution (TEE) for increasing mobile application security, we will not include them but only consider the software engineering techniques in this work. Also, resolving software engineering tasks often involves manual efforts, such as confirming the warnings yielded by static analyzers or labeling datasets for training machine learning models, etc. In this work, we will not take into account those manual approaches. For the remaining techniques, after discussing them among co-authors, we preliminarily categorize them as static-based, dynamic-based, and learning-based approaches. Fig. 8 highlights the represented ones. Static approaches are the analysis of programs performed without executing them. The widely used static approaches are listed in Fig. 8. These static approaches have been applied to the SE problems of mobile applications, Android frameworks, and mobile operating systems. Specifically, static approaches (e.g., taint analysis, symbolic execution, code instrumentation, model checking) are widely used to detect application bugs, including functional errors, code smells, security weaknesses/vulnerabilities, energy and performance bugs, permission escalations, etc. Beyond bug detection, static approaches (e.g., application hardening, code sign) are also used to increase the security and reliability of mobile applications. Moreover, with the rapid development of machine/deep learning, we have observed a trend to use static approaches to extract program features, which are then provided to learning approaches. Static Approaches. In contrast with static approaches, dynamic approaches are performed on programs during their execution. Similar to static approaches, dynamic approaches are also applied for program testing. Widely used dynamic testing techniques include search-based testing, black-box/random testing, grey-box fuzzing, concolic execution, event-driven test generation, mutation testing, etc. Dynamic program analysis is also applied for security analysis (e.g., dynamic taint analysis and runtime monitoring) and automated program repair. Dynamic approaches. Beyond the traditional static and dynamic approaches, we have seen an increasing trend that applies machine/deep learning techniques to solve mobile software engineering problems. Learning techniques train models by extracting features from large program artifacts and have achieved significant success in the field of code analysis. Learning-based techniques have been applied to solve many mobile software engineering tasks, including vulnerability detection, privacy issues detection, program testing, code smell checking, etc. Moreover, it has recently garnered considerable research attention to employ deep learning techniques to thwart Android malware attacks. Learning-based approaches [9] (𝑔11 OR ... OR 𝑔1𝑥 ) AND (𝑔21 OR ... OR 𝑔2𝑦), where 𝑔1𝑖 ∈ 𝐺1, 𝑔2𝑗 ∈ 𝐺2, and 1 ≤ 𝑖 ≤ 𝑥, 1 ≤ 𝑗 ≤ 𝑦, for which 𝑥 and 𝑦 are the number of keywords in G1 and G2, respectively [10] Indeed, app developers often cannot make profits directly from the apps per se as they are often made available to users as free apps. This paper is under CC 4.0 license. available on arxiv

Software Engineering for OpenHarmony: Overview of Mobile Software Engineering

Authors: (1) LI LI, Beihang University, China; (2) XIANG GAO, Beihang University, China; (3) HAILONG SUN, Beihang University, China; (4) CHUNMING HU, Beihang University, China; (5) XIAOYU SUN, The Australian National University, Australia; (6) HAOYU WANG, Huazhong University of Science and Technology, China; (7) HAIPENG CAI, Washington State University, Pullman, USA; (8) TING SU, East China Normal University, China; (9) XIAPU LUO, The Hong Kong Polytechnic University, China; (10) TEGAWENDÉ F. BISSYANDÉ, University of Luxembourg, Luxembourg; (11) JACQUES KLEIN, University of Luxembourg, Luxembourg; (12) JOHN GRUNDY, Monash University, Australia; (13) TAO XIE, Peking University, China; (14) HAIBO CHEN, Shanghai Jiao Tong University, China; (15) HUAIMIN WANG, National University of Defense Technology, China. Table of Links Introduction Background of OpenHarmony The State Of OpenHarmony Ecosystem Overview Of Mobile Software Engineering The Research Roadmap Discussion Related Work Conclusion & References 1 INTRODUCTION Mobile Software Engineering has been a hot topic for many years. It concerns all the aspects of software engineering in mobile, including the design, development, validation, execution, and evolution of mobile applications. This has been considered extremely important as nowadays our lives have been empowered by the massive increase in the use of mobile apps. Indeed, the number of mobile devices will reach 7 billion in 2023. The number of mobile apps that can be run on each mobile device (for both Android and iOS) has exceeded the 2 million mark. Furthermore, these figures are constantly increasing, thanks to app stores and marketplaces that allow users to effortlessly download and install applications. Mobile platforms are rapidly evolving as well in order to continuously integrate diverse and powerful capabilities, including various sensors, cameras, wireless communication channels, as well as on-device memory and disk capacities. As a result of ingeniously applying these technological developments, developers of mobile software are pushing the boundaries with innovative mobile services and exciting mobile applications. Consequently, due to the rapid development and evolution of mobile software, developers face new software engineering challenges. To address these challenges, researchers in the software engineering community have explored various research directions and developed lots of novel tools supported by formally grounded methods. Indeed, researchers have proposed various static program analysis approaches (i.e., by just scanning the code without actually running mobile apps) for characterizing issues (including ones related to mobile security, compatibility, energy consumption, etc.) of mobile apps [64]. For example, Arzt et al. [7] have designed and developed the famous FlowDroid approach that performs static taint analysis of Android apps for pinpointing privacy leaks. Except static analysis approaches, researchers have also invented various dynamic testing approaches (i.e., by actually running mobile apps on devices) for detecting potential defects of mobile apps at runtime [54]. For example, Amalfitano et al. [5] have proposed an GUI ripping approach for automated testing of Android apps. Su et al. [98] have proposed to achieve the same purpose through a model-based approach. The aforementioned research approaches have contributed to the huge success of the current flourishing mobile ecosystem, including both Android and iOS. Unfortunately, these approaches cannot directly benefit OpenHarmony[1] , which is a new opensourced mobile platform launched by the OpenAtom Foundation after receiving a donation of the open-source code from Huawei. These approaches, theoretically, should be generic and hence should also work for OpenHarmony. However, it still requires significant engineering efforts to achieve that due to the following reasons (more details will be given in the background section): (1) the Openharmony platform empowers a new framework supported by layered architecture, (2) Openharmony apps are written by a newly designed language called ArkTS. Unlike Android and iOS, which have been mature for many years and each has a healthy ecosystem to support their growth, the development of the Openharmony ecosystem is still at an earlier stage. We, therefore, argue that OpenHarmony requires more help from the software engineering research community.[2] We call on actions for conducting software engineering research for OpenHarmony. As our initial attempt, we decide to present to the community an initial research roadmap for guiding our mobile software engineering community to achieve that. We start by conducting a lightweight literature review of Mobile Software Engineering to understand the current achievements. We then conduct a comparative study to locate the technical gaps between mature platforms and OpenHarmony. Based on that, we summarize the technical deficiencies of OpenHarmony and propose a roadmap for our research community to complete. [1] https://www.openharmony.cn [2] This could be regarded as new opportunities for the mobile software engineering community This paper is under CC 4.0 license. available on arxiv

Software Engineering for OpenHarmony: A Research Roadmap

1. Forget about hardware; the real money is in software For months, headlines have fixated on Huawei's limitations in producing advanced 7nm chips, while Apple excels with its cutting-edge 3nm technology. However, this focus on hardware overlooks a critical lesson from history: software, not hardware, drives  profits. Bill Gates built the Microsoft empire not by building hardware—outsourced to IBM—but by dominating the software business. Today, Microsoft’s software business dwarfs IBM's historical combined hardware and software projects. Huawei’s HarmonyOS NEXT, already claimed to be the world’s third-largest mobile OS with over a billion installations, represents a significant milestone. While currently present mostly in China, the system has the potential to expand to other markets, particularly in the Global South. HarmonyOS could appeal to nations seeking to reduce reliance on U.S.-based technology. A large user base often translates into substantial revenue streams over time. Moreover, Huawei is working on extending its OS to personal computers under the name HarmonyOS PC, with a target release in Q3 2025. Even if this initiative gains traction solely within China, particularly in government systems, it could reduce the dominance of Microsoft Windows and Apple macOS, cutting into their lucrative revenue streams. 2. Know-How and Engineering Workforce Unlike hardware chip technology, software OS tools and expertise are widely accessible, starting with the open-source availability of Android OS. This means the necessary know-how is either readily available or easy to acquire. China also boasts a large skilled engineering workforce—reportedly four times larger than that of the U.S.—providing Huawei with ample talent at a competitive cost to develop its new OS. With sufficient time and resources, Huawei is likely to succeed in its HarmonyOS project. 3. Marketing or Reality? Reports suggest that HarmonyOS is facing some challenges. According to a media report, even the migration of China’s own WeChat app to the platform has encountered difficulties. This is not entirely surprising, as building a stable operating system is a highly complex and demanding task. Even Steve Jobs famously staged parts of his iPhone presentation during its early development stages. It required time and engineering effort for Apple to refine and perfect the product. Huawei faces a similar trajectory. If it commits to the project with enough talent, funding, and patience, HarmonyOS could mature into a serious alternative to Microsoft and Apple’s software dominance. 4 References [1] HarmonyOS [2] HarmonyOS NEXT [3] China's first fully home-grown mobile operating system HarmonyOS NEXT launched [4] https://www.huaweicentral.com/category/harmonyos/harmonyos-next/ [5] HarmonyOS NEXT becomes the world’s third major mobile OS, joining Android and iOS [6] Huawei’s HarmonyOS NEXT faces major test with WeChat: Will it pass? [7] Huawei’s China-Made 7nm Chip ‘Years Behind US’, Raimondo Says [8] In Huawei’s battle against Apple, chips are still a weakness, says analyst [9] HuaweiNext-gen Huawei PCs will ditch Windows for HarmonyOS: Chairman [10] Huawei HarmonyOS PC rumored to delay till Q3 2025 [11] HarmonyOS PC UI design allegedly surfaced on Huawei Developer website [12] The Global Distribution of STEM Graduates: Which Countries Lead the Way? [13] How Steve Jobs Faked His Way Through Unveiling the iPhone 1. Forget about hardware; the real money is in software For months, headlines have fixated on Huawei's limitations in producing advanced 7nm chips, while Apple excels with its cutting-edge 3nm technology. However, this focus on hardware overlooks a critical lesson from history: software, not hardware, drives  profits. Bill Gates built the Microsoft empire not by building hardware—outsourced to IBM—but by dominating the software business. Today, Microsoft’s software business dwarfs IBM's historical combined hardware and software projects. Huawei’s HarmonyOS NEXT , already claimed to be the world’s third-largest mobile OS with over a billion installations, represents a significant milestone. While currently present mostly in China, the system has the potential to expand to other markets, particularly in the Global South. HarmonyOS could appeal to nations seeking to reduce reliance on U.S.-based technology. A large user base often translates into substantial revenue streams over time. HarmonyOS NEXT world’s third-largest mobile OS Moreover, Huawei is working on extending its OS to personal computers under the name HarmonyOS PC , with a target release in Q3 2025 . Even if this initiative gains traction solely within China, particularly in government systems, it could reduce the dominance of Microsoft Windows and Apple macOS, cutting into their lucrative revenue streams. extending its OS to personal computers HarmonyOS PC Q3 2025 2. Know-How and Engineering Workforce Unlike hardware chip technology, software OS tools and expertise are widely accessible, starting with the open-source availability of Android OS. This means the necessary know-how is either readily available or easy to acquire. China also boasts a large skilled engineering workforce —reportedly four times larger than that of the U.S.—providing Huawei with ample talent at a competitive cost to develop its new OS. With sufficient time and resources, Huawei is likely to succeed in its HarmonyOS project. large skilled engineering workforce 3. Marketing or Reality? Reports suggest that HarmonyOS is facing some challenges. According to a media report, even the migration of China’s own WeChat app to the platform has encountered difficulties. This is not entirely surprising, as building a stable operating system is a highly complex and demanding task. Even Steve Jobs famously staged parts of his iPhone presentation during its early development stages. It required time and engineering effort for Apple to refine and perfect the product. Huawei faces a similar trajectory. If it commits to the project with enough talent, funding, and patience, HarmonyOS could mature into a serious alternative to Microsoft and Apple’s software dominance. 4 References [1] HarmonyOS HarmonyOS [2] HarmonyOS NEXT HarmonyOS NEXT [3] China's first fully home-grown mobile operating system HarmonyOS NEXT launched China's first fully home-grown mobile operating system HarmonyOS NEXT launched [4] https://www.huaweicentral.com/category/harmonyos/harmonyos-next/ https://www.huaweicentral.com/category/harmonyos/harmonyos-next/ [5] HarmonyOS NEXT becomes the world’s third major mobile OS, joining Android and iOS HarmonyOS NEXT becomes the world’s third major mobile OS, joining Android and iOS [6] Huawei’s HarmonyOS NEXT faces major test with WeChat: Will it pass? Huawei’s HarmonyOS NEXT faces major test with WeChat: Will it pass? [7] Huawei’s China-Made 7nm Chip ‘Years Behind US’, Raimondo Says Huawei’s China-Made 7nm Chip ‘Years Behind US’, Raimondo Says [8] In Huawei’s battle against Apple, chips are still a weakness, says analyst In Huawei’s battle against Apple, chips are still a weakness, says analyst [9] HuaweiNext-gen Huawei PCs will ditch Windows for HarmonyOS: Chairman HuaweiNext-gen Huawei PCs will ditch Windows for HarmonyOS: Chairman [10] Huawei HarmonyOS PC rumored to delay till Q3 2025 Huawei HarmonyOS PC rumored to delay till Q3 2025 [11] HarmonyOS PC UI design allegedly surfaced on Huawei Developer website HarmonyOS PC UI design allegedly surfaced on Huawei Developer website [12] The Global Distribution of STEM Graduates: Which Countries Lead the Way? The Global Distribution of STEM Graduates: Which Countries Lead the Way? [13] How Steve Jobs Faked His Way Through Unveiling the iPhone How Steve Jobs Faked His Way Through Unveiling the iPhone

Experts Skeptical of Huawei Silicon May Be Missing the Forest for the Trees

الخبراء المتشككون في السيليكون الخاص بشركة هواوي ربما يغفلون عن حقيقة الأمر

“Huawei Silicon”a şübhə ilə yanaşan ekspertlər ağaclar üçün meşədən məhrum ola bilərlər

Stručnjaci skeptični prema Huawei Silicijum možda nedostaje šuma za drveće

Los expertos escépticos sobre el silicio de Huawei podrían estar perdiendo de vista el bosque detrás de los árboles

מומחים הספקנים לגבי סיליקון Huawei עשויים להחמיץ את היער בשביל העצים

ファーウェイのシリコンに懐疑的な専門家は、木を見て森を見ずかもしれない

Pakar Ragu-ragu Huawei Silicon Mungkin Hilang Hutan untuk Pokok

Ditsebi tše di Belaelago Silicon ya Huawei e ka ba e Hlolwa Sethokgwa bakeng sa Dihlare

Стручњаци скептични према Хуавеи силицијуму можда недостаје шума за дрвеће

Hünärmenler Huawei Silikonyna şübheli garaýarlar, agaçlar üçin tokaýy ýitirip bilerler

Huawei Silikonuna Şüpheyle Yaklaşan Uzmanlar Ormanı Ağaçlardan Kaçırıyor Olabilir

Huawei kremniyiga shubha bilan qaraydigan ekspertlar daraxtlar uchun o‘rmon yetishmasligi mumkin

In software engineering, a data pipeline is exactly what it suggests: a way for data to ‘flow’ from a source to a destination. Every data pipeline consists of 3 stages. Extract phase: In this phase, data is extracted from the source. This could be hitting some API, reading files from a object storage (like AWS S3) or running queries on a database (like AWS RDS). Transform phase: In this phase, transformations are applied to the data to massage it into a different format. This could be transformations at rest (like normalizing all documents in a data lake) or transformations in motion (like converting from one class to another). Load phase: In this phase, the data is loaded into another storage. This could be a data warehouse, database or even a customer facing tool like a Tableau dashboard. Every data pipeline begins with the Extract phase. But the order in which the other 2 phases occur divides data pipelines into 2 broad categories, which are explained below. ETL Pipelines In ETL pipelines, the order of operations is Extract, Transform and Load. In ETL pipelines, data is extracted into from a single (or multiple) source(s), transformed on the fly using a set of business rules and loaded into a target repository. ETL pipelines usually move data into a relational store like a SQL database and this makes subsequent querying very fast. ELT Pipelines In ELT pipelines, the order of operations is Extract, Load and Transform. In ELT pipeline, data is extracted into from a single (or multiple) source(s). This raw data is stored in a data lake. Transformations are applied when required before the data is consumed or server. ELT is a relatively modern phenomenon, made possible by better technology in recent times. Advantages of ELT over ETL ELT, though new to the field, has some advantages over ETL that make it really useful in some scenarios. No need to formalize schema
This is a big advantage. ETL applications require all the transformations to be formalized and coded before the data can be ingested into the organization’s data stores. ELT has no such requirements. This means new data can be made available as soon as we know how to extract it.


Ingestion of unstructured data
ETL by definition can not deal with unstructured data. For transformations to take place consistently for every data point, we expect the data to have a certain predictable structure. This is not the case with ELT however. ELT pipelines are capable of handling both structured and unstructured data because they make no assumptions of any underlying structure.


Retention of Data from external sources
Many times, we may not know immediately how we want to consume data. We just know that we ‘may’ consume it later, and hence will definitely want to store it in a system controlled by us, like a data lake. ELT pipelines help us ingest raw data quickly into our data lake without wasting time thinking about how to transform them into a usable format. This is very useful for AI-based applications, especially. Conclusion ETL and ELT pipelines have their own advantages and disadvantages. While ETL pipelines are often the first preference, ELT pipelines could very well be more advantageous to your particular use case. The final decision, of course, depends on details and specifications required from the data pipeline. In software engineering, a data pipeline is exactly what it suggests: a way for data to ‘flow’ from a source to a destination. Every data pipeline consists of 3 stages. Extract phase: In this phase, data is extracted from the source. This could be hitting some API, reading files from a object storage (like AWS S3) or running queries on a database (like AWS RDS). Extract Transform phase: In this phase, transformations are applied to the data to massage it into a different format. This could be transformations at rest (like normalizing all documents in a data lake) or transformations in motion (like converting from one class to another). Transform Load phase: In this phase, the data is loaded into another storage. This could be a data warehouse, database or even a customer facing tool like a Tableau dashboard. Load Every data pipeline begins with the Extract phase. But the order in which the other 2 phases occur divides data pipelines into 2 broad categories, which are explained below. ETL Pipelines ETL Pipelines In ETL pipelines, the order of operations is E xtract, T ransform and L oad. E T L In ETL pipelines, data is extracted into from a single (or multiple) source(s), transformed on the fly using a set of business rules and loaded into a target repository. ETL pipelines usually move data into a relational store like a SQL database and this makes subsequent querying very fast. ELT Pipelines ELT Pipelines In ELT pipelines, the order of operations is E xtract, L oad and T ransform. E L T In ELT pipeline, data is extracted into from a single (or multiple) source(s). This raw data is stored in a data lake. Transformations are applied when required before the data is consumed or server. ELT is a relatively modern phenomenon, made possible by better technology in recent times. Advantages of ELT over ETL Advantages of ELT over ETL ELT, though new to the field, has some advantages over ETL that make it really useful in some scenarios. No need to formalize schema
This is a big advantage. ETL applications require all the transformations to be formalized and coded before the data can be ingested into the organization’s data stores. ELT has no such requirements. This means new data can be made available as soon as we know how to extract it. Ingestion of unstructured data
ETL by definition can not deal with unstructured data. For transformations to take place consistently for every data point, we expect the data to have a certain predictable structure. This is not the case with ELT however. ELT pipelines are capable of handling both structured and unstructured data because they make no assumptions of any underlying structure. Retention of Data from external sources
Many times, we may not know immediately how we want to consume data. We just know that we ‘may’ consume it later, and hence will definitely want to store it in a system controlled by us, like a data lake. ELT pipelines help us ingest raw data quickly into our data lake without wasting time thinking about how to transform them into a usable format. This is very useful for AI-based applications, especially. No need to formalize schema This is a big advantage. ETL applications require all the transformations to be formalized and coded before the data can be ingested into the organization’s data stores. ELT has no such requirements. This means new data can be made available as soon as we know how to extract it. No need to formalize schema No need to formalize schema This is a big advantage. ETL applications require all the transformations to be formalized and coded before the data can be ingested into the organization’s data stores. ELT has no such requirements. This means new data can be made available as soon as we know how to extract it. before Ingestion of unstructured data ETL by definition can not deal with unstructured data. For transformations to take place consistently for every data point, we expect the data to have a certain predictable structure. This is not the case with ELT however. ELT pipelines are capable of handling both structured and unstructured data because they make no assumptions of any underlying structure. Ingestion of unstructured data Ingestion of unstructured data ETL by definition can not deal with unstructured data. For transformations to take place consistently for every data point, we expect the data to have a certain predictable structure. This is not the case with ELT however. ELT pipelines are capable of handling both structured and unstructured data because they make no assumptions of any underlying structure. Retention of Data from external sources Many times, we may not know immediately how we want to consume data. We just know that we ‘may’ consume it later, and hence will definitely want to store it in a system controlled by us, like a data lake. ELT pipelines help us ingest raw data quickly into our data lake without wasting time thinking about how to transform them into a usable format. This is very useful for AI-based applications, especially. Retention of Data from external sources Retention of Data from external sources Many times, we may not know immediately how we want to consume data. We just know that we ‘may’ consume it later, and hence will definitely want to store it in a system controlled by us, like a data lake. ELT pipelines help us ingest raw data quickly into our data lake without wasting time thinking about how to transform them into a usable format. This is very useful for AI-based applications, especially. Conclusion ETL and ELT pipelines have their own advantages and disadvantages. While ETL pipelines are often the first preference, ELT pipelines could very well be more advantageous to your particular use case. The final decision, of course, depends on details and specifications required from the data pipeline.

Stories, News & Mentions about I3ENGINEERING

I3engineering’s Stories on HackerNoon

I3engineering's News & Mentions