Zettabytes

zettabytes

Are you data literate? An easy way to tell is to ask yourself whether you know how to source, analyze, and share quantitative information. If you don’t, you should learn how. The good news is the process is easy, even if you’re not a numbers person. Here’s what you need to do to improve your data literacy skills. What Is Data Literacy? Data literacy is the ability to interpret and interact with information meaningfully. You work with, perform operations on, and extract insights from it. Like regular literacy, there are multiple levels of competency, ranging from basic to adept. In other words, you don’t need to be an expert in the field to demonstrate an adequate understanding. What is poor data literacy? It is the inability to interpret, interact with, and understand information. If you work in an office or administrative setting, this skills gap might manifest as needing help deciphering graphs and reports. You may have difficulty distinguishing valuable insights from the noise. At the very least, you might struggle to understand analytics. For the record, you shouldn’t confuse data literacy with analytics. While analysis technically falls under the umbrella of being data literate, it’s only a tiny part of what you need to know. Arguably, uncovering patterns or correlations is the most useful thing you can do with a dataset, but it’s not the only thing. You shouldn’t feel discouraged if you’re not data literate yet. After all, even today’s high-ranking professionals aren’t. Only half of C-suite executives feel confident in their data literacy skills, with 45% admitting they make decisions based on gut feelings instead of facts. What Are Data Literacy Skills? While dozens of data literacy skills exist, you only need a handful to start your educational journey. Mastering the basics helps you build a solid knowledge foundation. 1. Critical Thinking Critical thinking is arguably one of the most important skills because there’s simply too much data floating around, much of which is inaccurate, outdated, or irrelevant. Knowing what questions to ask can help you determine whether you want to use a source, reference a website, or perform operations on a dataset. 2. Problem-Solving Sometimes, data is incomplete or messy — it doesn’t always fit into spreadsheets neatly. Problem-solving skills can help you make those metrics workable. While you’ll have to rely on mathematics often, you don’t need to excel in the subject. You don’t necessarily even need to know how to perform calculations. More often than not, knowing how to get to the answer is enough. Say you’re trying to find how much one figure increased over time to determine growth. Instead of memorizing a formula, you’d simply use a percentage change calculator. Creatively approaching problems can help you interact with data meaningfully. 3. Communication Communication is one of the most essential skills to learn because all worthwhile findings are reviewed and replicated. These processes can help you identify mistakes and uncover opportunities. At work, viewing information as a shared resource can help foster a culture of collaboration and communication. 4. Visualization Visualization is similar to communication in that it helps you get your point across. The key difference is you prioritize storytelling. Graphs, charts, and tables make plain metrics more interesting. They also make complex topics easier to comprehend — numbers seem more real when accompanied by informative illustrations. Why Data Literacy Matters Data literacy isn’t a term you come across daily, so it might not seem important. In reality, you could consider it a life skill since you interact with information all day, every day. Utility bills, news reports, ingredient labels, medication dosages, online polls, and product recommendations are good examples. You may even read reports and process spreadsheets at work. These skills can help you better understand the world because you live in the digital age. Even if you don’t realize it, you absorb quantitative data daily. Knowing how to interpret, analyze, and explain it can help you turn seemingly meaningless numbers into meaningful insights. You learn how to develop arguments, follow along at work, and spot misinformation better. Interestingly, possessing data literacy skills also makes it easier to keep up with the latest technology trends. You need a solid understanding to get involved in trends like artificial intelligence and the Internet of Things. Recognizing the importance of sourcing, dataset cleaning and analytics goes a long way. Frankly, becoming data literate is important because there’s too much information. Experts estimate the total amount created, copied and consumed will go from 64.2 zettabytes in 2020 to 180 zettabytes in 2025 — a 180% increase in just five years. Knowing what makes metrics relevant, accurate, and reliable is essential. What You Need to Understand About Data Before you can become data literate, there are a few fundamentals you must understand. How It Works There are multiple data types, including datasets, physical documents, studies, databases and records. At its most basic, it is broken into two categories — qualitative or quantitative. Whether you leverage numbers or descriptions, you’re working with information. In data science, data is often either described as structured or unstructured. The first fits neatly into spreadsheet tables, often containing numbers, dates or words. The latter is messier. It can be social media posts, audio clips, emails, videos or documents. Where It Goes You often have to collect data to extract any meaningful results. While tons of information is available online, it may not be recent, accurate or specific enough to be useful to you. Common sources are surveys, records, studies and analytics. A few decades ago, information was usually physical — people stored records in filing cabinets and documents in briefcases. Now, almost everything is digitally stored on hard drives or kept in the cloud for remote access. What It Does You can do a lot to a dataset, including processing, analysis and insight extraction. These use cases are probably the main reason your employer collects information, unless they’re a financial or health care institution and are required to do so by law. Before you can do anything with data, you need to clean it, which involves eliminating outliers, filling in missing fields and transforming any mistyped figures. It’s a lengthy process. According to one survey, data scientists spend 39% of their time on preparation and cleaning. How to Become Data Literate Where should you go to improve your data literacy? Since just under half of employees say their employer has offered data training courses, there may not be many opportunities at work. Thankfully, you don’t need predefined courses to develop the core skills. The best way to learn is to practice — even if you’re inexperienced. Consider getting a dataset online to practice on. If you don’t know what to use, try the weather. It makes for good practice because there are always recent, accurate forecasts. At first, your goal should be to evaluate the raw numbers to ensure you understand what’s in front of you. Then, you can transition to cleaning, performing analysis and extracting insights. Figure out what questions you want to ask. Is the change from the first day to the last significant? What percentage of days were overcast? If you feel like taking things a step further, try analyzing two datasets at once — one on weather and another on beachgoers. Does the ultraviolet index affect how busy the beach is? While that example may seem overly simplistic, analysis often is. Whether you’re trying to extract insights on employee retention, construction material cost, return fraud or the housing market, you can uncover patterns by asking straightforward questions and plugging numbers into the correct formulas. Your practice isn’t finished when you produce your findings. At that point, you proceed to visualization. The best way to get better at reading graphs and tables is to make your own. This way, you’ll get used to the fundamentals, allowing you to instantly recognize a visual’s shared characteristics when someone else presents information. Making Yourself Data Literate You won’t improve your data literacy skills overnight. However, even though it will likely be time-consuming, it doesn’t have to be complicated. To make the process more enjoyable, work with information that interests you. There are patterns, trends and insights to uncover in every industry, and there is a wealth of information online for free. Are you data literate? An easy way to tell is to ask yourself whether you know how to source, analyze, and share quantitative information. If you don’t, you should learn how. Are you data literate? An easy way to tell is to ask yourself whether you know how to source, analyze, and share quantitative information. If you don’t, you should learn how. The good news is the process is easy, even if you’re not a numbers person. Here’s what you need to do to improve your data literacy skills. What Is Data Literacy? Data literacy is the ability to interpret and interact with information meaningfully. You work with, perform operations on, and extract insights from it. Like regular literacy, there are multiple levels of competency, ranging from basic to adept. In other words, you don’t need to be an expert in the field to demonstrate an adequate understanding. What is poor data literacy? It is the inability to interpret, interact with, and understand information. If you work in an office or administrative setting, this skills gap might manifest as needing help deciphering graphs and reports. You may have difficulty distinguishing valuable insights from the noise. At the very least, you might struggle to understand analytics. For the record, you shouldn’t confuse data literacy with analytics. While analysis technically falls under the umbrella of being data literate, it’s only a tiny part of what you need to know. Arguably, uncovering patterns or correlations is the most useful thing you can do with a dataset, but it’s not the only thing. You shouldn’t feel discouraged if you’re not data literate yet. After all, even today’s high-ranking professionals aren’t. Only half of C-suite executives feel confident in their data literacy skills, with 45% admitting they make decisions based on gut feelings instead of facts. half of C-suite executives half of C-suite executives What Are Data Literacy Skills? While dozens of data literacy skills exist, you only need a handful to start your educational journey. Mastering the basics helps you build a solid knowledge foundation. 1. Critical Thinking Critical thinking is arguably one of the most important skills because there’s simply too much data floating around, much of which is inaccurate, outdated, or irrelevant. Knowing what questions to ask can help you determine whether you want to use a source, reference a website, or perform operations on a dataset. 2. Problem-Solving Sometimes, data is incomplete or messy — it doesn’t always fit into spreadsheets neatly. Problem-solving skills can help you make those metrics workable. While you’ll have to rely on mathematics often, you don’t need to excel in the subject. You don’t necessarily even need to know how to perform calculations. More often than not, knowing how to get to the answer is enough. Say you’re trying to find how much one figure increased over time to determine growth. Instead of memorizing a formula, you’d simply use a percentage change calculator. Creatively approaching problems can help you interact with data meaningfully. 3. Communication Communication is one of the most essential skills to learn because all worthwhile findings are reviewed and replicated. These processes can help you identify mistakes and uncover opportunities. At work, viewing information as a shared resource can help foster a culture of collaboration and communication. can help foster a culture can help foster a culture 4. Visualization Visualization is similar to communication in that it helps you get your point across. The key difference is you prioritize storytelling. Graphs, charts, and tables make plain metrics more interesting. They also make complex topics easier to comprehend — numbers seem more real when accompanied by informative illustrations. Why Data Literacy Matters Data literacy isn’t a term you come across daily, so it might not seem important. In reality, you could consider it a life skill since you interact with information all day, every day. Utility bills, news reports, ingredient labels, medication dosages, online polls, and product recommendations are good examples. You may even read reports and process spreadsheets at work. These skills can help you better understand the world because you live in the digital age. Even if you don’t realize it, you absorb quantitative data daily. Knowing how to interpret, analyze, and explain it can help you turn seemingly meaningless numbers into meaningful insights. You learn how to develop arguments, follow along at work, and spot misinformation better. help you better understand the world help you better understand the world Interestingly, possessing data literacy skills also makes it easier to keep up with the latest technology trends. You need a solid understanding to get involved in trends like artificial intelligence and the Internet of Things. Recognizing the importance of sourcing, dataset cleaning and analytics goes a long way. Frankly, becoming data literate is important because there’s too much information. Experts estimate the total amount created, copied and consumed will go from 64.2 zettabytes in 2020 to 180 zettabytes in 2025 — a 180% increase in just five years. Knowing what makes metrics relevant, accurate, and reliable is essential. from 64.2 zettabytes in 2020 from 64.2 zettabytes in 2020 What You Need to Understand About Data Before you can become data literate, there are a few fundamentals you must understand. How It Works There are multiple data types, including datasets, physical documents, studies, databases and records. At its most basic, it is broken into two categories — qualitative or quantitative. Whether you leverage numbers or descriptions, you’re working with information. In data science, data is often either described as structured or unstructured. The first fits neatly into spreadsheet tables, often containing numbers, dates or words. The latter is messier. It can be social media posts, audio clips, emails, videos or documents. Where It Goes You often have to collect data to extract any meaningful results. While tons of information is available online, it may not be recent, accurate or specific enough to be useful to you. Common sources are surveys, records, studies and analytics. A few decades ago, information was usually physical — people stored records in filing cabinets and documents in briefcases. Now, almost everything is digitally stored on hard drives or kept in the cloud for remote access. What It Does You can do a lot to a dataset, including processing, analysis and insight extraction. These use cases are probably the main reason your employer collects information, unless they’re a financial or health care institution and are required to do so by law. Before you can do anything with data, you need to clean it, which involves eliminating outliers, filling in missing fields and transforming any mistyped figures. It’s a lengthy process. According to one survey, data scientists spend 39% of their time on preparation and cleaning. spend 39% of their time spend 39% of their time How to Become Data Literate Where should you go to improve your data literacy? Since just under half of employees say their employer has offered data training courses, there may not be many opportunities at work. Thankfully, you don’t need predefined courses to develop the core skills. The best way to learn is to practice — even if you’re inexperienced. just under half of employees just under half of employees Consider getting a dataset online to practice on. If you don’t know what to use, try the weather. It makes for good practice because there are always recent, accurate forecasts. At first, your goal should be to evaluate the raw numbers to ensure you understand what’s in front of you. Then, you can transition to cleaning, performing analysis and extracting insights. Figure out what questions you want to ask. Is the change from the first day to the last significant? What percentage of days were overcast? If you feel like taking things a step further, try analyzing two datasets at once — one on weather and another on beachgoers. Does the ultraviolet index affect how busy the beach is? While that example may seem overly simplistic, analysis often is. Whether you’re trying to extract insights on employee retention, construction material cost, return fraud or the housing market, you can uncover patterns by asking straightforward questions and plugging numbers into the correct formulas. Your practice isn’t finished when you produce your findings. At that point, you proceed to visualization. The best way to get better at reading graphs and tables is to make your own. This way, you’ll get used to the fundamentals, allowing you to instantly recognize a visual’s shared characteristics when someone else presents information. Making Yourself Data Literate You won’t improve your data literacy skills overnight. However, even though it will likely be time-consuming, it doesn’t have to be complicated. To make the process more enjoyable, work with information that interests you. There are patterns, trends and insights to uncover in every industry, and there is a wealth of information online for free.

How to Improve Your Data Literacy Skills

የእርስዎን የውሂብ ማንበብ ችሎታዎች እንዴት ማሻሻል እንደሚችሉ

كيفية تحسين مهاراتك في مجال محو الأمية البيانات

Kunjamas Datos Alfabetización Yatiñanakax juk’amp askiptaspa

Məlumat Savadlılığı Bacarıqlarınızı Necə Təkmilləşdirə bilərsiniz

Cómo mejorar sus habilidades de alfabetización de datos

Hogyan fejlesztheti adatműveltségi készségeit

データリテラシースキルを向上させる方法

Tsela ya go Kaonafatša Mabokgoni a Gago a go Bala le go Ngwala ka Datha

Как улучшить свои навыки работы с данными

Maitiro eKuvandudza Yako Data Literacy Unyanzvi

ከመይ ጌርና ናይ ዳታ ምልላይ ክእለትና ነመሓይሽ

Ma'lumotlar savodxonligi ko'nikmalaringizni qanday yaxshilash mumkin

Architecting Scalable Databases for Large-Scale Systems

Highly Efficient and Secure Data Analysis Using Dask and AWS Best Practices

The world relies on data. It's the fuel that drives innovation and impacts lives. With the advent of the generative AI era, it's crucial for data engineers to position themselves as leaders and innovators in this exciting time. Statista predicted that the global data creation will explode a significant data volume between 2010 and 2035, from 2 zettabytes to an impressive volume of 2,142 zettabytes by 2025! Are you ready as data engineers for the coming year? This article explores 3 topics: Advanced Data Processing Techniques
Enhancing Data Security with AI-Driven Solutions
Ethical AI and Responsible Data Management Advanced Data Processing Techniques As data engineers, we must adapt to the rapidly evolving landscape of data science and AI. In 2025, there will be a plethora of data engineering tools and technologies to master. Artificial intelligence has given rise to many fields, including neural networks and deep learning. Generative AI trends are impossible without these foundational technologies. Thanks to the incredible power of AI, machines can now learn complex patterns and generate new data based on existing datasets. Embracing these advancements will be crucial for staying ahead in the data-driven world. Where Should Data Engineers Focus? Deep Learning: Data engineers have a plethora of tools to master in the realm of deep learning. Among the most well-known are TensorFlow and PyTorch. These frameworks enable data engineers to build and optimize neural networks for a variety of tasks, including natural language generation and image synthesis.
Neural Networks: Proficiency in designing, training, and fine-tuning neural networks is essential for data engineers. Mastery of these skills allows engineers to achieve remarkable results in a world where data is the driving force.
Real-World Solutions: Generative AI has the potential to revolutionize various industries. While it’s a powerful technology, there is always room for improvement. With practical applications across healthcare, finance, and education, generative AI can create transformative solutions that impact these sectors profoundly. By following the steps data engineers will make a huge difference in 2025 for more impact. Enhancing Data Security with AI-Driven Solutions Where There is Data, There is a Risk of Theft! With numerous solutions entering the market, it's crucial to secure data with AI-driven security measures. The frequency of cyber attacks is ever-increasing, and we must be prepared daily to protect our data. In the generative AI era, anything can happen. Here are some key aspects data engineers should focus on: Anomaly Detection: Data engineers don't have to do everything manually. Thanks to AI algorithms, they can detect unusual patterns and behaviors in data, identifying potential security threats. It's essential for data engineers to implement and master these algorithms to enhance their organization's security posture.
Automated Threat Response: AI-driven systems can automate threat response, significantly reducing the time required to mitigate risks manually. Leveraging AI for automated security solutions is crucial for efficient and effective threat management.
Data Encryption: AI is now a key ingredient for success in many fields. By integrating AI into robust encryption methods, we can prevent data theft and enhance overall security. Implementing AI-driven data encryption techniques is vital for protecting sensitive information. By following the key considerations, data engineers will make a huge difference in the industry. Ethical AI and Responsible Data Management AI is Ineffective Without Better Regulations! We need robust frameworks to enhance the AI ecosystem, ensuring trust and security for users. Data engineers should learn and adopt best practices that foster transparency and accountability, contributing to a thriving digital society. Here are the steps they should follow: Ethical AI Frameworks and Development Speed: Keeping pace with rapid AI development is challenging; frameworks must evolve to ensure alignment. Data engineers must acquaint themselves with leading industry frameworks to promote transparency, uphold human rights, and mitigate bias effectively.


Data Anonymization and GDPR Compliance: The GDPR framework goes beyond rhetoric! Services implementing GDPR guidelines enable effective information concealment, safeguarding user privacy comprehensively. GDPR-compliant services are pivotal for protecting sensitive data.


Transparency in AI Decision-Making: Users increasingly demand insights into decision-making processes of AI models. Establishing transparency not only fosters trust but also enhances credibility with stakeholders.
Conclusion “Data as Innovation Fuel” Data serves as the cornerstone for driving innovation. Simply building, optimizing, and managing complex systems isn't sufficient for today's data engineers. They must take on pivotal roles in fostering transparency and embracing continuous learning to cultivate a digital society that respects human rights and thrives. The world relies on data. It's the fuel that drives innovation and impacts lives. With the advent of the generative AI era, it's crucial for data engineers to position themselves as leaders and innovators in this exciting time. Statista predicted that the global data creation will explode a significant data volume between 2010 and 2035, from 2 zettabytes to an impressive volume of 2,142 zettabytes by 2025! Statista Are you ready as data engineers for the coming year? This article explores 3 topics: Advanced Data Processing Techniques Enhancing Data Security with AI-Driven Solutions Ethical AI and Responsible Data Management Advanced Data Processing Techniques Enhancing Data Security with AI-Driven Solutions Ethical AI and Responsible Data Management Advanced Data Processing Techniques As data engineers, we must adapt to the rapidly evolving landscape of data science and AI. In 2025, there will be a plethora of data engineering tools and technologies to master. Artificial intelligence has given rise to many fields, including neural networks and deep learning. Generative AI trends are impossible without these foundational technologies. Thanks to the incredible power of AI, machines can now learn complex patterns and generate new data based on existing datasets. Embracing these advancements will be crucial for staying ahead in the data-driven world. Where Should Data Engineers Focus? Where Should Data Engineers Focus? Deep Learning: Data engineers have a plethora of tools to master in the realm of deep learning. Among the most well-known are TensorFlow and PyTorch. These frameworks enable data engineers to build and optimize neural networks for a variety of tasks, including natural language generation and image synthesis. Neural Networks: Proficiency in designing, training, and fine-tuning neural networks is essential for data engineers. Mastery of these skills allows engineers to achieve remarkable results in a world where data is the driving force. Real-World Solutions: Generative AI has the potential to revolutionize various industries. While it’s a powerful technology, there is always room for improvement. With practical applications across healthcare, finance, and education, generative AI can create transformative solutions that impact these sectors profoundly. Deep Learning : Data engineers have a plethora of tools to master in the realm of deep learning. Among the most well-known are TensorFlow and PyTorch. These frameworks enable data engineers to build and optimize neural networks for a variety of tasks, including natural language generation and image synthesis. Deep Learning Neural Networks : Proficiency in designing, training, and fine-tuning neural networks is essential for data engineers. Mastery of these skills allows engineers to achieve remarkable results in a world where data is the driving force. Neural Networks Real-World Solutions : Generative AI has the potential to revolutionize various industries. While it’s a powerful technology, there is always room for improvement. With practical applications across healthcare, finance, and education, generative AI can create transformative solutions that impact these sectors profoundly. Real-World Solutions By following the steps data engineers will make a huge difference in 2025 for more impact. Enhancing Data Security with AI-Driven Solutions Enhancing Data Security with AI-Driven Solutions Where There is Data, There is a Risk of Theft! Where There is Data, There is a Risk of Theft! With numerous solutions entering the market, it's crucial to secure data with AI-driven security measures. The frequency of cyber attacks is ever-increasing, and we must be prepared daily to protect our data. In the generative AI era, anything can happen. Here are some key aspects data engineers should focus on: Anomaly Detection: Data engineers don't have to do everything manually. Thanks to AI algorithms, they can detect unusual patterns and behaviors in data, identifying potential security threats. It's essential for data engineers to implement and master these algorithms to enhance their organization's security posture. Automated Threat Response: AI-driven systems can automate threat response, significantly reducing the time required to mitigate risks manually. Leveraging AI for automated security solutions is crucial for efficient and effective threat management. Data Encryption: AI is now a key ingredient for success in many fields. By integrating AI into robust encryption methods, we can prevent data theft and enhance overall security. Implementing AI-driven data encryption techniques is vital for protecting sensitive information. Anomaly Detection : Data engineers don't have to do everything manually. Thanks to AI algorithms, they can detect unusual patterns and behaviors in data, identifying potential security threats. It's essential for data engineers to implement and master these algorithms to enhance their organization's security posture. Anomaly Detection Automated Threat Response : AI-driven systems can automate threat response, significantly reducing the time required to mitigate risks manually. Leveraging AI for automated security solutions is crucial for efficient and effective threat management. Automated Threat Response Data Encryption : AI is now a key ingredient for success in many fields. By integrating AI into robust encryption methods, we can prevent data theft and enhance overall security. Implementing AI-driven data encryption techniques is vital for protecting sensitive information. Data Encryption By following the key considerations, data engineers will make a huge difference in the industry. Ethical AI and Responsible Data Management Ethical AI and Responsible Data Management AI is Ineffective Without Better Regulations! AI is Ineffective Without Better Regulations! We need robust frameworks to enhance the AI ecosystem, ensuring trust and security for users. Data engineers should learn and adopt best practices that foster transparency and accountability, contributing to a thriving digital society. Here are the steps they should follow: Ethical AI Frameworks and Development Speed: Keeping pace with rapid AI development is challenging; frameworks must evolve to ensure alignment. Data engineers must acquaint themselves with leading industry frameworks to promote transparency, uphold human rights, and mitigate bias effectively. Data Anonymization and GDPR Compliance: The GDPR framework goes beyond rhetoric! Services implementing GDPR guidelines enable effective information concealment, safeguarding user privacy comprehensively. GDPR-compliant services are pivotal for protecting sensitive data. Transparency in AI Decision-Making: Users increasingly demand insights into decision-making processes of AI models. Establishing transparency not only fosters trust but also enhances credibility with stakeholders.
Conclusion Ethical AI Frameworks and Development Speed: Keeping pace with rapid AI development is challenging; frameworks must evolve to ensure alignment. Data engineers must acquaint themselves with leading industry frameworks to promote transparency, uphold human rights, and mitigate bias effectively. Ethical AI Frameworks and Development Speed: Keeping pace with rapid AI development is challenging; frameworks must evolve to ensure alignment. Data engineers must acquaint themselves with leading industry frameworks to promote transparency, uphold human rights, and mitigate bias effectively. Ethical AI Frameworks and Development Speed: Data Anonymization and GDPR Compliance: The GDPR framework goes beyond rhetoric! Services implementing GDPR guidelines enable effective information concealment, safeguarding user privacy comprehensively. GDPR-compliant services are pivotal for protecting sensitive data. Data Anonymization and GDPR Compliance: The GDPR framework goes beyond rhetoric! Services implementing GDPR guidelines enable effective information concealment, safeguarding user privacy comprehensively. GDPR-compliant services are pivotal for protecting sensitive data. Data Anonymization and GDPR Compliance: Transparency in AI Decision-Making: Users increasingly demand insights into decision-making processes of AI models. Establishing transparency not only fosters trust but also enhances credibility with stakeholders. Conclusion Transparency in AI Decision-Making: Users increasingly demand insights into decision-making processes of AI models. Establishing transparency not only fosters trust but also enhances credibility with stakeholders. Transparency in AI Decision-Making: Conclusion “Data as Innovation Fuel” Data serves as the cornerstone for driving innovation. Simply building, optimizing, and managing complex systems isn't sufficient for today's data engineers. They must take on pivotal roles in fostering transparency and embracing continuous learning to cultivate a digital society that respects human rights and thrives. “Data as Innovation Fuel”

Generative AI: 3 Topics to Learn as a Data Engineer in 2024 and Beyond

Why Is the AI Surge Boosting HDD Stocks? Finis Conner Says: Drives Are to AI as Batteries Are to EVs

In the age of AI, the field of traditional ‘data science’ might seem antiquated or not as important, innovative, or lucrative as emerging technologies. However, the ebb and flow of tech trends are underpinned by traditional data science methodologies, which means data scientists are still a vital component in the tech ecosystem and will become ubiquitous across all industries as organizations increasingly rely on data to fuel their business objectives. Arguably, even more so thanks to the advent of generative AI, which has revolutionized our ability to digest vast volumes of data at the touch of a button, but still needs data scientists to interpret and verify AI-gleaned insights into accurate, and actionable assessments. McKinsey’s latest Global Survey has identified that inaccuracy poses the biggest threat to companies using AI and that investment in analytical AI is now on par with investment in generative AI. 3 data scientist jobs that pay over $100k Data Scientist, Advertising, Spotify, New York ($107,766 - $153,951 a year)
Principal Data Scientist, Salesforce, San Francisco ($185,800 - $296,400 a year)
Director - Data Scientist Prudential, Newark ($173,500 - $234,700) Looking to the future According to a recent study, the data science platform market is forecast to grow to a valuation of $322.9 billion by 2026 at a CAGR of 27.7% during the period of 2021 to 2026. Understandable when you take into consideration that 147 zettabytes of data has been created so far this year, compared to 120 zettabytes in 2023 and 97 zettabytes in 2022 (it’s estimated that each person can potentially generate around 15.87 terabytes of data daily). The even better news is that whether you’re a data analyst or data engineer, your skillset and ability to enforce data policies and shape how AI is used is commanding high salaries across the board. Unlocking your potential So, how can you pivot to a career in data if you’re already proficient in other areas of tech and are now ready to make the transition mid-career? For starters, having an understanding of Python, R, and SQL is essential for a career in data science as it will allow you to not only automate tasks but manage structured data, aggregate data, and perform tasks related to machine learning and data visualization. 3 high-paying data scientist jobs Data Scientist, Booz Allen, Chantilly ($75,600 - $172,000 a year)
AML Data Scientist II, TD Bank, Mount Laurel ($95,264 - $155,376)
Applied NLP Data Scientist, ubs, New York ($150,000 - $200,000 a year) If you need to brush up on one or all of the above, Udemy is offering a concise (and free) course on data science, machine learning, data analysis, data visualization using Python and R. Or to get a better understanding of how you can develop your machine learning skills, Andrew Ng, the co-founder of Google Brain and an adjunct professor at Stanford University is running a series of short courses via Coursera. Whatever you choose, one thing is clear: data science remains one of the most appealing (and generously-paid) career routes in tech. Ready to find your next role in tech? Whether you want to pivot to data science or put your coding expertise to good use, visit the Hackernoon Job Board today By Aoibhinn Mc Bride In the age of AI , the field of traditional ‘data science’ might seem antiquated or not as important, innovative, or lucrative as emerging technologies. AI However, the ebb and flow of tech trends are underpinned by traditional data science methodologies, which means data scientists are still a vital component in the tech ecosystem and will become ubiquitous across all industries as organizations increasingly rely on data to fuel their business objectives. Arguably, even more so thanks to the advent of generative AI, which has revolutionized our ability to digest vast volumes of data at the touch of a button, but still needs data scientists to interpret and verify AI-gleaned insights into accurate, and actionable assessments. McKinsey’s latest Global Survey has identified that inaccuracy poses the biggest threat to companies using AI and that investment in analytical AI is now on par with investment in generative AI. 3 data scientist jobs that pay over $100k Data Scientist, Advertising, Spotify, New York ($107,766 - $153,951 a year) Principal Data Scientist, Salesforce, San Francisco ($185,800 - $296,400 a year) Director - Data Scientist Prudential, Newark ($173,500 - $234,700) Data Scientist, Advertising, Spotify, New York ($107,766 - $153,951 a year) Data Scientist, Advertising, Spotify, New York ($107,766 - $153,951 a year) Principal Data Scientist, Salesforce, San Francisco ($185,800 - $296,400 a year) Principal Data Scientist, Salesforce, San Francisco ($185,800 - $296,400 a year) Director - Data Scientist Prudential, Newark ($173,500 - $234,700) Director - Data Scientist Prudential, Newark ($173,500 - $234,700) Looking to the future According to a recent study, the data science platform market is forecast to grow to a valuation of $322.9 billion by 2026 at a CAGR of 27.7% during the period of 2021 to 2026. Understandable when you take into consideration that 147 zettabytes of data has been created so far this year, compared to 120 zettabytes in 2023 and 97 zettabytes in 2022 (it’s estimated that each person can potentially generate around 15.87 terabytes of data daily). The even better news is that whether you’re a data analyst or data engineer, your skillset and ability to enforce data policies and shape how AI is used is commanding high salaries across the board. Unlocking your potential So, how can you pivot to a career in data if you’re already proficient in other areas of tech and are now ready to make the transition mid-career? For starters, having an understanding of Python, R, and SQL is essential for a career in data science as it will allow you to not only automate tasks but manage structured data, aggregate data, and perform tasks related to machine learning and data visualization. 3 high-paying data scientist jobs Data Scientist, Booz Allen, Chantilly ($75,600 - $172,000 a year) AML Data Scientist II, TD Bank, Mount Laurel ($95,264 - $155,376) Applied NLP Data Scientist, ubs, New York ($150,000 - $200,000 a year) Data Scientist, Booz Allen, Chantilly ($75,600 - $172,000 a year) Data Scientist, Booz Allen, Chantilly ($75,600 - $172,000 a year) AML Data Scientist II, TD Bank, Mount Laurel ($95,264 - $155,376) AML Data Scientist II, TD Bank, Mount Laurel ($95,264 - $155,376) Applied NLP Data Scientist, ubs, New York ($150,000 - $200,000 a year) Applied NLP Data Scientist, ubs, New York ($150,000 - $200,000 a year) If you need to brush up on one or all of the above, Udemy is offering a concise (and free) course on data science, machine learning, data analysis, data visualization using Python and R. Or to get a better understanding of how you can develop your machine learning skills, Andrew Ng, the co-founder of Google Brain and an adjunct professor at Stanford University is running a series of short courses via Coursera. Whatever you choose, one thing is clear: data science remains one of the most appealing (and generously-paid) career routes in tech. Ready to find your next role in tech? Whether you want to pivot to data science or put your coding expertise to good use, visit the Hackernoon Job Board today Ready to find your next role in tech? Whether you want to pivot to data science or put your coding expertise to good use, visit the Hackernoon Job Board today Ready to find your next role in tech? Whether you want to pivot to data science or put your coding expertise to good use, visit the Hackernoon Job Board today By Aoibhinn Mc Bride By Aoibhinn Mc Bride

Want To Earn 100k and Above? Then Look to Data Science Jobs

Artificial Intelligence has been advancing at an unprecedented rate, causing fear about whether AI will dominate various aspects of our lives in the near future and “take our jobs.” In this article, by leveraging Moore's Law and other technological growth principles, I’m going to make an educated estimation about the state of AI in the coming years. Let’s explore these predictions and explain why AI may not completely take over within the next five years, and what the timeline might look like for AI to reach a truly advanced state. The Current State of AI AI technology has made significant strides in recent years. From natural language processing to computer vision, AI applications are becoming increasingly sophisticated, whether that’s the recent GPT-4o model, Google’s Gemini, or any one of the models in the race to become the most capable AI model. There are lots of people using these models in positive ways to unlock new methodologies in research and education, but there are also lots of people using these technologies for evil such as deepfakes. However, despite these advancements, there are still considerable challenges and limitations that need to be addressed. Current AI systems need massive amounts of data, computational power, and energy to function effectively. Moore's Law and Its Implications for AI Moore's Law, the observation that the number of transistors on a microchip doubles approximately every two years, has been a reliable predictor of technological progress for decades. More generally, Moore’s Law points to technology getting twice as accessible, cheaper, and efficient every two years. According to Moore's Law, we can expect computational power to continue increasing exponentially. However, there are physical and economic limits to this growth that may impact the pace of AI development. Estimating AI's Growth Using Known Laws By extrapolating from Moore's Law and considering other technological growth trends, we can make predictions about AI's capabilities in the coming years. Computational Power If we consider top-of-the-notch GPUs today, let’s say NVIDIA A100, it provides approximately 312 teraflops (TFLOPS) of performance. Moore's Law suggests a doubling of transistors every two years, which implies a potential 8-fold increase in computational power over five years (2^2.5 ≈ 5.66). Future GPU Performance = 312 TFLOPS × 5.66 ≈ 1766 TFLOPS However, the physical limitations of silicon and the economic costs of producing ever-smaller transistors could slow this growth. The increase might realistically be closer to 4 times due to these constraints: Realistic Future GPU Performance=312 TFLOPS×4≈1248 TFLOPS Data Availability and Quality According to projections by IDC (International Data Corporation), the size of the global datasphere will increase from 64.2 zettabytes in 2020 to 175 zettabytes by 2025. It is essential for the training of increasingly complex AI models that data is growing exponentially. But there are still a lot of obstacles to overcome, including concerns about data privacy, data quality, and effectively labeling and processing this data.The availability of high-quality, labeled data is critical, even with an increase in data output. Preprocessing and data labeling are obstacles that will not scale linearly with data increase, according to current trends. Energy Consumption AI training models like GPT-3 require substantial energy. GPT-3 used approximately 1,287 MWh to train. GPT-4, developed approximately 3 years after its predecessor required over 50 gigawatt-hours, so around 50 times the amount it took to train GPT-3. Assuming AI models become more complex and require more data and computational power, energy consumption could grow significantly. If we estimate a similar 50x increase or a 5000% increase in complexity to around 2500 gigawatt-hours for the next model that could be released in the next 5 years, this would require large improvements in energy production and efficiency. Given current energy efficiency improvements of around 10% per year (source: Lawrence Livermore National Laboratory), this only translates to approximately a 60% increase in energy efficiencies over a 5-year period if the energy production technologies don’t improve significantly. The Need for Increased Complexity in AI Models Current AI models struggle with many major issues today. Lack of Up-to-date Information: Current models are trained on static datasets and do not have real-time access to the internet for the most up-to-date information. Hallucinations: AI models sometimes generate information that is plausible-sounding but incorrect or nonsensical. Understanding Context and Nuance: Handling nuanced human language, context, and intent remains a challenge for AI. To address these issues, future AI models will need to incorporate: Real-Time Data Integration: Access to and processing of real-time data from the internet to provide up-to-date responses. Improved Model Architectures: Enhancements in model architectures to reduce hallucinations and improve the reliability of generated content. Contextual Understanding: Advanced natural language processing techniques to better understand and maintain context over long conversations. Increased Model Complexity Solving these problems will significantly increase the complexity of AI models. For example: Real-Time Data Integration: Requires constant data updates and real-time processing capabilities.
Improved Model Architectures: Likely to involve larger and more complex models with more parameters.
Contextual Understanding: Demands more sophisticated training algorithms and potentially longer training times. Let's assume each of these improvements independently doubles the complexity of the AI model. If we start with the current complexity (C), integrating these advancements might lead to: Future Complexity=C×2(real-time data)×2(improved architectures)×2(contextual understanding)=C×2^3 =C×8 Let’s estimate a 5x increase in complexity as a more conservative yet substantial figure. Data Quality and Ethics AI systems depend heavily on high-quality, unbiased data for training and decision-making. While the quantity of data is increasing rapidly, the quality and ethical considerations surrounding this data are equally crucial. Data Generation and Processing With the increase from 64.2 to 175 zettabytes, data generation will likely outpace the current capabilities of processing and labeling. Ensuring data quality and addressing biases require significant resources. Let's consider the human labor and computational resources needed for data labeling. Assuming 1 hour of human labor is needed to label 1 GB of data, labeling 175 zettabytes would require: 175 ZB×1 hour/GB×10^9  GB/ZB =175×10^9  hours This is not feasible with current resources, indicating a substantial gap in data quality and ethical processing. Conclusion: When Will AI Be in a Truly Advanced State In the Next, Say 5 Years? The idea that AI will "take over" in the next five years seems improbable, even if it will surely continue to advance and become more deeply integrated into many industries. There are several obstacles to overcome, including issues with energy usage, data quality, and processing power limitations. AI will not be able to evolve as fast without significant advancements in energy efficiency, data ethics, and technology.It seems more reasonable to anticipate that AI will become truly advanced within the next ten to fifteen years, based on existing patterns and reasonable development estimates. This prolonged period enables the development and successful implementation of essential ethical, technological, and resource-based breakthroughs. Sources and References: Moore, G. E. (1965). Cramming more components onto integrated circuits. Electronics, 38(8), 114-117.
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.
Markoff, J. (2015). Machines of Loving Grace: The Quest for Common Ground Between Humans and Robots. HarperCollins.
IDC. (2020). Global DataSphere Forecast 2021-2025.
Lawrence Livermore National Laboratory. (2021). Energy Efficiency Trends in Computing. Artificial Intelligence has been advancing at an unprecedented rate, causing fear about whether AI will dominate various aspects of our lives in the near future and “take our jobs.” In this article, by leveraging Moore's Law and other technological growth principles, I’m going to make an educated estimation about the state of AI in the coming years. Let’s explore these predictions and explain why AI may not completely take over within the next five years, and what the timeline might look like for AI to reach a truly advanced state. The Current State of AI AI technology has made significant strides in recent years. From natural language processing to computer vision, AI applications are becoming increasingly sophisticated, whether that’s the recent GPT-4o model, Google’s Gemini, or any one of the models in the race to become the most capable AI model. There are lots of people using these models in positive ways to unlock new methodologies in research and education, but there are also lots of people using these technologies for evil such as deepfakes. However, despite these advancements, there are still considerable challenges and limitations that need to be addressed. Current AI systems need massive amounts of data, computational power, and energy to function effectively. Moore's Law and Its Implications for AI Moore's Law , the observation that the number of transistors on a microchip doubles approximately every two years, has been a reliable predictor of technological progress for decades. More generally, Moore’s Law points to technology getting twice as accessible, cheaper, and efficient every two years. According to Moore's Law, we can expect computational power to continue increasing exponentially. Moore's Law However, there are physical and economic limits to this growth that may impact the pace of AI development. Estimating AI's Growth Using Known Laws By extrapolating from Moore's Law and considering other technological growth trends, we can make predictions about AI's capabilities in the coming years. Computational Power If we consider top-of-the-notch GPUs today, let’s say NVIDIA A100, it provides approximately 312 teraflops (TFLOPS) of performance. Moore's Law suggests a doubling of transistors every two years, which implies a potential 8-fold increase in computational power over five years (2^2.5 ≈ 5.66). Future GPU Performance = 312 TFLOPS × 5.66 ≈ 1766 TFLOPS However, the physical limitations of silicon and the economic costs of producing ever-smaller transistors could slow this growth. The increase might realistically be closer to 4 times due to these constraints: Realistic Future GPU Performance=312 TFLOPS×4≈1248 TFLOPS Data Availability and Quality According to projections by IDC (International Data Corporation), the size of the global datasphere will increase from 64.2 zettabytes in 2020 to 175 zettabytes by 2025. It is essential for the training of increasingly complex AI models that data is growing exponentially. But there are still a lot of obstacles to overcome, including concerns about data privacy, data quality, and effectively labeling and processing this data. The availability of high-quality, labeled data is critical, even with an increase in data output. Preprocessing and data labeling are obstacles that will not scale linearly with data increase, according to current trends. Energy Consumption AI training models like GPT-3 require substantial energy. GPT-3 used approximately 1,287 MWh to train. GPT-4, developed approximately 3 years after its predecessor required over 50 gigawatt-hours , so around 50 times the amount it took to train GPT-3. Assuming AI models become more complex and require more data and computational power, energy consumption could grow significantly. 50 gigawatt-hours If we estimate a similar 50x increase or a 5000% increase in complexity to around 2500 gigawatt-hours for the next model that could be released in the next 5 years, this would require large improvements in energy production and efficiency. 2500 gigawatt-hours Given current energy efficiency improvements of around 10% per year (source: Lawrence Livermore National Laboratory), this only translates to approximately a 60% increase in energy efficiencies over a 5-year period if the energy production technologies don’t improve significantly. The Need for Increased Complexity in AI Models Current AI models struggle with many major issues today. Lack of Up-to-date Information: Current models are trained on static datasets and do not have real-time access to the internet for the most up-to-date information. Lack of Up-to-date Information: Hallucinations: AI models sometimes generate information that is plausible-sounding but incorrect or nonsensical. Hallucinations: Understanding Context and Nuance: Handling nuanced human language, context, and intent remains a challenge for AI. Understanding Context and Nuance: To address these issues, future AI models will need to incorporate: Real-Time Data Integration: Access to and processing of real-time data from the internet to provide up-to-date responses. Real-Time Data Integration: Access to and processing of real-time data from the internet to provide up-to-date responses. Real-Time Data Integration: Improved Model Architectures: Enhancements in model architectures to reduce hallucinations and improve the reliability of generated content. Improved Model Architectures: Enhancements in model architectures to reduce hallucinations and improve the reliability of generated content. Improved Model Architectures: Contextual Understanding: Advanced natural language processing techniques to better understand and maintain context over long conversations. Contextual Understanding: Advanced natural language processing techniques to better understand and maintain context over long conversations. Contextual Understanding: Increased Model Complexity Increased Model Complexity Solving these problems will significantly increase the complexity of AI models. For example: Real-Time Data Integration: Requires constant data updates and real-time processing capabilities. Improved Model Architectures: Likely to involve larger and more complex models with more parameters. Contextual Understanding: Demands more sophisticated training algorithms and potentially longer training times. Real-Time Data Integration: Requires constant data updates and real-time processing capabilities. Real-Time Data Integration: Improved Model Architectures: Likely to involve larger and more complex models with more parameters. Improved Model Architectures: Contextual Understanding: Demands more sophisticated training algorithms and potentially longer training times. Contextual Understanding: Let's assume each of these improvements independently doubles the complexity of the AI model. If we start with the current complexity (C), integrating these advancements might lead to: Future Complexity=C×2(real-time data)×2(improved architectures)×2(contextual understanding)=C×2^3 =C×8 Future Complexity=C×2(real-time data)×2(improved architectures)×2(contextual understanding)=C×2^3 =C×8 Let’s estimate a 5x increase in complexity as a more conservative yet substantial figure. Let’s estimate a 5x increase in complexity as a more conservative yet substantial figure. Data Quality and Ethics AI systems depend heavily on high-quality, unbiased data for training and decision-making. While the quantity of data is increasing rapidly, the quality and ethical considerations surrounding this data are equally crucial. Data Generation and Processing With the increase from 64.2 to 175 zettabytes, data generation will likely outpace the current capabilities of processing and labeling. Ensuring data quality and addressing biases require significant resources. Let's consider the human labor and computational resources needed for data labeling. Assuming 1 hour of human labor is needed to label 1 GB of data, labeling 175 zettabytes would require: 175 ZB×1 hour/GB×10^9  GB/ZB =175×10^9  hours 175 ZB×1 hour/GB×10^9  GB/ZB = 175×10^9  hours 175×10^9  hours This is not feasible with current resources, indicating a substantial gap in data quality and ethical processing. Conclusion: When Will AI Be in a Truly Advanced State In the Next, Say 5 Years? The idea that AI will "take over" in the next five years seems improbable, even if it will surely continue to advance and become more deeply integrated into many industries. There are several obstacles to overcome, including issues with energy usage, data quality, and processing power limitations. AI will not be able to evolve as fast without significant advancements in energy efficiency, data ethics, and technology. It seems more reasonable to anticipate that AI will become truly advanced within the next ten to fifteen years, based on existing patterns and reasonable development estimates. This prolonged period enables the development and successful implementation of essential ethical, technological, and resource-based breakthroughs. Sources and References : Sources and References Moore, G. E. (1965). Cramming more components onto integrated circuits. Electronics, 38(8), 114-117. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. Markoff, J. (2015). Machines of Loving Grace: The Quest for Common Ground Between Humans and Robots. HarperCollins. IDC. (2020). Global DataSphere Forecast 2021-2025. Lawrence Livermore National Laboratory. (2021). Energy Efficiency Trends in Computing. Moore, G. E. (1965). Cramming more components onto integrated circuits. Electronics , 38(8), 114-117. Electronics Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning . MIT Press. Deep Learning Markoff, J. (2015). Machines of Loving Grace: The Quest for Common Ground Between Humans and Robots. HarperCollins . HarperCollins IDC. (2020). Global DataSphere Forecast 2021-2025. Lawrence Livermore National Laboratory. (2021). Energy Efficiency Trends in Computing.

Predicting the Future of AI With Moore's Law: When Will It Really Take Over?

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Hatem A. Alharbi, CSchool of Electronic and Electrical Engineering, University of Leeds, LS2 9JT, United Kingdom;
(2) Taisir E.H. Elgorashi, School of Electronic and Electrical Engineering, University of Leeds, LS2 9JT, United Kingdom;
(3) Jaafar M.H. Elmirghani, School of Electronic and Electrical Engineering, University of Leeds, LS2 9JT, United Kingdom. Table of Links Abstract & Introduction Related Works Repealing Net Neutrality Profit-Driven Model Results Conclusions & References Biographies Abstract Network neutrality (net neutrality) is the principle of treating equally all Internet traffic regardless of its source, destination, content, application or other related distinguishing metrics. Under net neutrality, ISPs are compelled to charge all content providers (CPs) the same per Gbps rate despite the growing profit achieved by CPs. In this paper, we study the impact of the repeal of net neutrality on communication networks by developing a techno-economic Mixed Integer Linear Programming (MILP) model to maximize the potential profit ISPs can achieve by offering their services to CPs. We focus on video delivery as video traffic accounts for 78% of the cloud traffic. We consider an ISP that offers CPs different classes of service representing typical video content qualities including standard definition (SD), high definition (HD) and ultra-high definition (UHD) video. The MILP model maximizes the ISP profit by optimizing the prices of the different classes according to the users demand sensitivity to the change in price, referred to as Price Elasticity of Demand (PED). We analyze how PED impacts the profit in different CP delivery scenarios in cloud-fog architectures. The results show that the repeal of net neutrality can potentially increase ISPs profit by a factor of 8 with a pricing scheme that discriminates against data intensive content. Also, the repeal of net neutrality positively impacts the network energy efficiency by reducing the core network power consumption by 55% as a result of suppressing data intensive content compared to the net neutrality scenario. Index Terms— Net neutrality, AT&T, IP over WDM networks, profit, power consumption. I. INTRODUCTION Network (net) neutrality regulations prohibit ISPs from applying different treatment to IP packets based on their content e.g. prioritizing, blocking or throttling certain Internet content or allowing quality differentiation. Net neutrality, which was scrapped by the US Federal Communications Commission (FCC) in December 2017, has been the subject of remarkable debate in recent years between ISPs and CPs with each side trying to exploit their assets and expand their profit and influence. The debate is fueled by the rapidly escalating demand for CPs services as a result of the interconnection between Internet and broadcasting markets. Cisco forecasts [1] that by 2021, annual global Internet traffic will hit 2.2 Zettabytes per month and CPs datacenters will be the source of 71% of this traffic. Online video services are the primary cause of this accelerated growth in Internet traffic. Video streaming is poised to consume 78% of the total CPs bandwidth with 75% of Internet video traffic originating from higher video services quality (HD and UHD). Proponents of preferential treatment of Internet traffic complain that the increasing demand for data-intensive content creates a significant burden on the communication network. They argue that removing net neutrality will give ISPs further control of their infrastructure, which is crucial in order to improve QoS and reduce security threats. Another argument is that a significant fraction of the profit of this tremendously growing market is seized by CPs whereas ISPs act as a transit or transport medium into CPs customers. In the US, the quarterly profit margin of AT&T (an ISP) has been almost stable over the last six years whereas Netflix (a CP) profit margin has risen up in rapid pace from 0.7% to 9.8% within the same period [2], [3]. In contrast, advocates warn that removing net neutrality will slow down the innovation in the Internet and its content and will limit the content competition by disadvantaging small businesses, and subsequently, diminish online services. Deploying traffic discrimination in video delivery services has many challenges, e.g. detecting video packets and enforcing a policy on a certain video quality. Traffic discrimination in IP communication networks has been surveyed intensively in the literature. Several traffic management practices have been surveyed in [4]. The authors highlighted that traffic discrimination taxonomy has four features: (i) characteristics or condition of the traffic (e.g. based on content, protocol or source/destination); (ii) traffic classification (e.g. based on flow rate, header information or routing); (iii) mechanism of discrimination (e.g. modify, delay, drop or block); and (iv) perceived discrimination by end-users. Video traffic can be analyzed using two mechanisms; deep packet inspection (DPI) [5], [6] or traffic profiling [7], [8]. DPI examines the data packets that are sent over the network and traffic profiling detects abnormal network traffic by comparing new traffic against previous traffic profile. For example, an alarm can be triggered if the data rate transmitted over the network (measured in bps) spikes above the desired data rate, which could indicate an increase in data rate. QoS for video services delivery can be applied either by reserving network bandwidth for video packets (e.g. using IntServ) or labelling video content as high priority e.g. by applying Differentiated Services (e.g. using DiffServ) [9]. The Internet ecosystem is complex with many stakeholders. As illustrated in Fig. 1, the main stakeholders in the Internet ecosystem are; ISPs, CPs, content delivery networks (CDNs) and end-users. Users pay ISPs a subscription fee to get Internet access and subscribe to CPs (if required) to access their content. CPs subscribe to a CDN to access storage and processing capacity and to deliver their content to customers. CDNs are responsible for sending CPs content at large scale over ISPs network infrastructure, e.g. the CP Netflix collaborates with the CDN Amazon Web Services (AWS) to reach their customers [10]. ISPs play as the key intermediary in the delivery process as they provide the required connectivity between users and content. Most ISPs such as AT&T [11] and Comcast [12] are now providing CDN services in additional to networking services. To simplify our analysis, we consider a direct relationship between ISP and CPs. Due to net neutrality regulations, current pricing policy of ISP networking services applies a fixed charge which is not linked with bitrate usage. For example, in the US, AT&T uses a fixed pricing model by charging CPs $3,282 per 10 Gbps per month [13] regardless of the content type transferred to users (either UHD video content or a simple text message). The rest of this paper is organized as follows: Section II briefly summarizes related work. We describe the pricing scheme we used in this paper and the profit-driven model we adopted in Section III. Our results are presented in Section IV. In Section V, we provide concluding remarks. This paper is available on arxiv under CC 4.0 license. Authors: (1) Hatem A. Alharbi, CSchool of Electronic and Electrical Engineering, University of Leeds, LS2 9JT, United Kingdom; (2) Taisir E.H. Elgorashi, School of Electronic and Electrical Engineering, University of Leeds, LS2 9JT, United Kingdom; (3) Jaafar M.H. Elmirghani, School of Electronic and Electrical Engineering, University of Leeds, LS2 9JT, United Kingdom. This paper is available on arxiv under CC 4.0 license. available on arxiv Authors: Authors: (1) Hatem A. Alharbi, CSchool of Electronic and Electrical Engineering, University of Leeds, LS2 9JT, United Kingdom; (2) Taisir E.H. Elgorashi, School of Electronic and Electrical Engineering, University of Leeds, LS2 9JT, United Kingdom; (3) Jaafar M.H. Elmirghani, School of Electronic and Electrical Engineering, University of Leeds, LS2 9JT, United Kingdom. Table of Links Abstract & Introduction Abstract & Introduction Related Works Related Works Repealing Net Neutrality Repealing Net Neutrality Profit-Driven Model Results Profit-Driven Model Results Conclusions & References Conclusions & References Biographies Biographies Abstract Network neutrality (net neutrality) is the principle of treating equally all Internet traffic regardless of its source, destination, content, application or other related distinguishing metrics. Under net neutrality, ISPs are compelled to charge all content providers (CPs) the same per Gbps rate despite the growing profit achieved by CPs. In this paper, we study the impact of the repeal of net neutrality on communication networks by developing a techno-economic Mixed Integer Linear Programming (MILP) model to maximize the potential profit ISPs can achieve by offering their services to CPs. We focus on video delivery as video traffic accounts for 78% of the cloud traffic. We consider an ISP that offers CPs different classes of service representing typical video content qualities including standard definition (SD), high definition (HD) and ultra-high definition (UHD) video. The MILP model maximizes the ISP profit by optimizing the prices of the different classes according to the users demand sensitivity to the change in price, referred to as Price Elasticity of Demand (PED). We analyze how PED impacts the profit in different CP delivery scenarios in cloud-fog architectures. The results show that the repeal of net neutrality can potentially increase ISPs profit by a factor of 8 with a pricing scheme that discriminates against data intensive content. Also, the repeal of net neutrality positively impacts the network energy efficiency by reducing the core network power consumption by 55% as a result of suppressing data intensive content compared to the net neutrality scenario. Index Terms — Net neutrality, AT&T, IP over WDM networks, profit, power consumption. Index Terms I. INTRODUCTION Network (net) neutrality regulations prohibit ISPs from applying different treatment to IP packets based on their content e.g. prioritizing, blocking or throttling certain Internet content or allowing quality differentiation. Net neutrality, which was scrapped by the US Federal Communications Commission (FCC) in December 2017, has been the subject of remarkable debate in recent years between ISPs and CPs with each side trying to exploit their assets and expand their profit and influence. The debate is fueled by the rapidly escalating demand for CPs services as a result of the interconnection between Internet and broadcasting markets. Cisco forecasts [1] that by 2021, annual global Internet traffic will hit 2.2 Zettabytes per month and CPs datacenters will be the source of 71% of this traffic. Online video services are the primary cause of this accelerated growth in Internet traffic. Video streaming is poised to consume 78% of the total CPs bandwidth with 75% of Internet video traffic originating from higher video services quality (HD and UHD). Proponents of preferential treatment of Internet traffic complain that the increasing demand for data-intensive content creates a significant burden on the communication network. They argue that removing net neutrality will give ISPs further control of their infrastructure, which is crucial in order to improve QoS and reduce security threats. Another argument is that a significant fraction of the profit of this tremendously growing market is seized by CPs whereas ISPs act as a transit or transport medium into CPs customers. In the US, the quarterly profit margin of AT&T (an ISP) has been almost stable over the last six years whereas Netflix (a CP) profit margin has risen up in rapid pace from 0.7% to 9.8% within the same period [2], [3]. In contrast, advocates warn that removing net neutrality will slow down the innovation in the Internet and its content and will limit the content competition by disadvantaging small businesses, and subsequently, diminish online services. Deploying traffic discrimination in video delivery services has many challenges, e.g. detecting video packets and enforcing a policy on a certain video quality. Traffic discrimination in IP communication networks has been surveyed intensively in the literature. Several traffic management practices have been surveyed in [4]. The authors highlighted that traffic discrimination taxonomy has four features: (i) characteristics or condition of the traffic (e.g. based on content, protocol or source/destination); (ii) traffic classification (e.g. based on flow rate, header information or routing); (iii) mechanism of discrimination (e.g. modify, delay, drop or block); and (iv) perceived discrimination by end-users. Video traffic can be analyzed using two mechanisms; deep packet inspection (DPI) [5], [6] or traffic profiling [7], [8]. DPI examines the data packets that are sent over the network and traffic profiling detects abnormal network traffic by comparing new traffic against previous traffic profile. For example, an alarm can be triggered if the data rate transmitted over the network (measured in bps) spikes above the desired data rate, which could indicate an increase in data rate. QoS for video services delivery can be applied either by reserving network bandwidth for video packets (e.g. using IntServ) or labelling video content as high priority e.g. by applying Differentiated Services (e.g. using DiffServ) [9]. The Internet ecosystem is complex with many stakeholders. As illustrated in Fig. 1, the main stakeholders in the Internet ecosystem are; ISPs, CPs, content delivery networks (CDNs) and end-users. Users pay ISPs a subscription fee to get Internet access and subscribe to CPs (if required) to access their content. CPs subscribe to a CDN to access storage and processing capacity and to deliver their content to customers. CDNs are responsible for sending CPs content at large scale over ISPs network infrastructure, e.g. the CP Netflix collaborates with the CDN Amazon Web Services (AWS) to reach their customers [10]. ISPs play as the key intermediary in the delivery process as they provide the required connectivity between users and content. Most ISPs such as AT&T [11] and Comcast [12] are now providing CDN services in additional to networking services. To simplify our analysis, we consider a direct relationship between ISP and CPs. Due to net neutrality regulations, current pricing policy of ISP networking services applies a fixed charge which is not linked with bitrate usage. For example, in the US, AT&T uses a fixed pricing model by charging CPs $3,282 per 10 Gbps per month [13] regardless of the content type transferred to users (either UHD video content or a simple text message). The rest of this paper is organized as follows: Section II briefly summarizes related work. We describe the pricing scheme we used in this paper and the profit-driven model we adopted in Section III. Our results are presented in Section IV. In Section V, we provide concluding remarks.

Impact of the Net Neutrality Repeal on Communication Networks: Abstract & Introduction

This paper is available on arxiv under CC 4.0 license. Authors: (1) Vogt, Lars, TIB Leibniz Information Centre for Science and Technology; (2) Konrad, Marcel, TIB Leibniz Information Centre for Science and Technology; (3) Prinz, Manuel, TIB Leibniz Information Centre for Science and Technology. Table of Links Abstract & Introduction Interoperability Semantic interoperability and what natural languages like English can teach us Requirements for successfully communicating terms and statements Parallels between the structure of natural language statements and data schemata with implications for semantic interoperability What makes a term a good term and a schema a good schema? The need for a machine-actionable Rosetta Stone for (meta)data that acts as an interlingua for specifying reference terms and reference schemata to support cognitive and semantic interoperability Rosetta Stone and machine-readability: UPRIs, XML Schema datatypes, and RDF for communicating terms, datatypes, and statements Rosetta Stone and machine-interpretability: Wikidata and a modeling paradigm for (meta)data statements based on English Rosetta Stone and semantic interoperability: Specifying term mappings and schema crosswalks Rosetta Stone and cognitive interoperability: Specifying display templates and using a query builder Discussion Related work Conclusion, Acknowledgements, & References Abstract In order to effectively manage the overwhelming influx of data, it is crucial to ensure that data is findable, accessible, interoperable, and reusable (FAIR). While ontologies and knowledge graphs have been employed to enhance FAIRness, challenges remain regarding semantic and cognitive interoperability. We explore how English facilitates reliable communication of terms and statements, and transfer our findings to a framework of ontologies and knowledge graphs, while treating terms and statements as minimal information units. We categorize statement types based on their predicates, recognizing the limitations of modeling non-binary predicates with multiple triples, which negatively impacts interoperability. Terms are associated with different frames of reference, and different operations require different schemata. Term mappings and schema crosswalks are therefore vital for semantic interoperability. We propose a machine-actionable Rosetta Stone Framework for (meta)data, which uses reference terms and schemata as an interlingua to minimize mappings and crosswalks. Modeling statements rather than a human-independent reality ensures cognitive familiarity and thus better interoperability of data structures. We extend this Rosetta modeling paradigm to reference schemata, resulting in simple schemata with a consistent structure across statement types, empowering domain experts to create their own schemata using the Rosetta Editor, without requiring knowledge of semantics. The Editor also allows specifying textual and graphical display templates for each schema, delivering human-readable data representations alongside machine-actionable data structures. The Rosetta Query Builder derives queries based on completed input forms and the information from corresponding reference schemata. This work sets the conceptual ground for the Rosetta Stone Framework that we plan to develop in the future. Introduction In the current era, we are witnessing an exponential growth in the generation and consumption of data. Statista’s projections from 2020 indicate that the global data creation, capture, copying, and consumption will reach an estimated 97 zettabytes in 2022, with a staggering daily volume of approximately 328.77 billion terabytes by 2023, reflecting a notable 23.71% increase over the preceding year, with videos accounting for over 50% of the global traffic (1,2). Additionally, data production in the last two years accounted for a striking 90% of the entire data generated worldwide in 2016 (3), illustrating the rapid pace of data expansion. The doubling of the overall data volume every three years (4) further accentuates the magnitude of this exponential growth. Simultaneously, the scholarly domain is witnessing a surge in publications, with an annual output of over 7 million academic papers (5). These figures emphasize the urgency of harnessing machine support, as without it, the sheer volume of (meta)data threatens to overwhelm and impede meaningful insights. With this in mind, it is essential to facilitate machine-actionable (meta)data in scientific research, so that machines assist researchers in identifying relevant (meta)data pertaining to a specific research question. Moreover, enhancing the machine-actionability of (meta)data offers a potential solution to the reproducibility crisis in science (6), enabling the availability, findability, and usability of raw (meta)data (7). Considering the substantial global investment in research and development, which reached a staggering sum of $1,609,214 billion in 2021 (8), failure to render the corresponding output of (meta)data machine-actionable could potentially result in redundant research efforts. By enabling machine-actionable (meta)data, substantial savings can be achieved, redirecting resources towards novel and non-repetitive research endeavors. However, the concept of machine-actionable (meta)data requires clarification. Looking at the definition of machine-actionable (see Box 1), it is evident that this attribute cannot be simplified as a mere Boolean property. Instead, it exists on a spectrum, allowing for degrees of machine-actionability. Numerous operations can potentially be applied to a given set of (meta)data, and the ability to apply even a single operation would suffice to classify the (meta)data as machine-actionable. Consequently, specifying the set of operations that can be applied to the (meta)data, along with the corresponding tools or code, is more meaningful than labeling the entire set as machine-actionable. For instance, one could state that “Dataset A is machine-actionable with respect to operation X using tool Y”. It is worth noting that reading a dataset could be considered an operation itself. Therefore, datasets documented in formats as PDF, XML, or even ASCII files can be considered machine-readable and, to some extent, already machine-actionable. Moreover, if a dataset is machine-readable, search operations can be performed on it, enabling the identification of specific elements through label matching, for example. The success of search operations serves as a measure of the findability of (meta)data. Similarly to machine-actionability, findability cannot be characterized as a Boolean property. Machine-readable (meta)data can be found through label matching, while interpretable (meta)data can be found through their meaning, referent, or contextual information. Thus, (meta)data that are readable but not interpretable possess limited findability. It is important to emphasize that our definition of machine-actionability, as outlined in Box 1, strictly depends on machine-interoperability. Consequently, machine-reading of a dataset and machine-searching based on label matching are not considered proper examples for operations that fulfill the requirements of machine-actionability. In 2016, the were introduced, providing a framework to assess the extent to which (meta)data are Findable, Accessible, Interoperable, and Reusable for both machines and humans alike (11). These principles have gained increasing attention from the research, industry, and knowledge management tool development communities in recent years (11–16). Furthermore, stakeholders in science and research policy have recognized the significance of the FAIR Principles. The economic impact of FAIR research (meta)data was estimated by the European Union (EU) in 2018, revealing that the lack of FAIR (meta)data costs the EU economy at least 10.2 billion Euros annually. Taking into account the positive effects of FAIR (meta)data on data quality and machine-readability, an additional 16 billion Euros were estimated (17). Consequently, the High Level Expert Group of the European Open Science Cloud (EOSC) recommended the establishment of an Internet of FAIR Data and Services (IFDS) (18). The IFDS aims to enable data-rich institutions, research projects, and citizen-science initiatives to make their (meta)data accessible in accordance with the FAIR Guiding Principles, while retaining control over ethical, privacy, and legal aspects of their (meta)data (following Barend Mons’ data visiting as opposed to data sharing (19)). Achieving this goal requires the provision of rich machine-actionable (meta)data, their organization into FAIR Digital Objects (20,21), each identifiable by a Unique Persistent and Resolvable Identifier (UPRI), and the development of suitable concepts and tools for human-readable interface outputs and search capabilities. Although progress has been made toward building the IFDS (see the GO FAIR Initiative and EOSC), the current state of the FAIRness of (meta)data in many data-rich institutions and companies is still far from ideal. FAIR Guiding Principles for (meta)data The increasing volume, velocity, variety, and complexity of data present significant challenges that traditional methods and techniques for handling, processing, analyzing, managing, storing, and retrieving (meta)data struggle to address effectively within a reasonable timeframe (22). However, knowledge graphs in conjunction with ontologies, offer a promising technical solution for implementing the FAIR Guiding Principles, thanks to their transparent semantics, highly structured syntax, and standardized formats (23,24). Knowledge graphs represent instances, classes, and their relationships as resources with their own UPRIs. These UPRIs are employed to denote relationships between entities using the triple syntax of Subject-Predicate-Object. Each particular relationship is thus modeled as a structured set of three distinct data points. In contrast, relational databases model entity relationships between data table columns and not between individual data points. Consequently, knowledge graphs outperform relational databases in handling complex queries on densely connected data, which is often the case with research (meta)data. Therefore, knowledge graphs are particularly well suited for FAIR research and development, as well as any task requiring efficient and detailed retrieval of (meta)data. Nonetheless, employing a knowledge graph to document (meta)data does not guarantee adherence to FAIR Principles. Achieving FAIRness necessitates additional guidelines, such as consistent usage of the same semantic data model for identical types of (meta)data statements to ensure schematic interoperability (10), as well as organizing (meta)data into FAIR Digital Objects (20,21). Knowledge graphs, being a relatively new concept and technology, introduce their own specific technical, conceptual, and societal challenges. This is evident in the somewhat ambiguous nature of the knowledge graph concept (23) and the absence of commonly accepted standards, given the diverse technical and conceptual incarnations ranging from labeled property graphs like Neo4J to approaches based on the Resource Description Framework (RDF), employing RDF stores and applications of description logic using the Web Ontology Language (OWL). Regardless of these considerations, it can be argued that the demand for FAIR and machine-actionable (meta)data presupposes their During such communication processes, preserving the meaning and reference of the message between sender and receiver is crucial, requiring both parties to share the same background knowledge, encompassing lexical competences, syntax and grammar rules, and relevant contextual knowledge. successful communication between machines and between humans and machines. In this paper, we lay the conceptual groundwork for a future machine-actionable Rosetta Stone Framework designed to achieve cognitive and semantic interoperability of (meta)data. We begin by examining the notion of interoperability and emphasizing the necessity of cognitive interoperability. Drawing inspiration from natural languages like English, we explore how semantic interoperability can be understood by analyzing the way terms and statements convey meaning and information. Recognizing terms and statements as basic units of information, we investigate the linguistic structures that ensure reliable information communication and draw parallels to the structures found in (meta)data schemata. This analysis yields insights into achieving semantic interoperability of terms and statements. Our main argument centers around the requirement for a machine-actionable Rosetta Stone Framework for (meta)data, which addresses terminological and propositional interoperability. This framework leverages reference terms and reference schemata as an interlingua to establish term mappings and schema crosswalks, facilitating cognitive and semantic interoperability. The emphasis lies on a modeling paradigm that enables machine-interpretability of (meta)data, prioritizing their findability and interoperability over reasoning capabilities. This prioritization opens up new avenues for modeling by shifting away from the paradigm frequently applied in science that focuses on modeling a human-independent reality. Instead, the Rosetta Modeling Paradigm models the structure of statements to enhance efficient and reliable information communication. We propose two alternative versions of a Rosetta Modeling Approach: a minimal light version and a full version that supports versioning of statements and tracks the editing history of each statement within a knowledge graph. Furthermore, we discuss the specifications for a Rosetta Editor, a tool that will allow domain experts to create new reference schemata without requiring expertise in semantics or computer science. The Rosetta Editor will also enable the specification of display templates for human-readable text-based and graph-based representation of (meta)data. Additionally, we propose a Rosetta Query Builder that allows users to write queries without the need for a graph query language. Finally, we provide an overview of the Rosetta Framework and explore related work in the field before presenting our concluding remarks. It is important to note that this paper represents conceptual work and serves as a foundation for future development, testing, and implementation of the envisioned Rosetta Stone Framework.

Towards a Rosetta Stone for (meta)data: Abstract & Introduction

306 Stories To Learn About Artificial Intelligence

Stories, News & Mentions about ZETTABYTES

Zettabytes’s Stories on HackerNoon