Data is the most powerful asset in the world today. As fantastic as it may seem, data is more valuable than anything in the world today and the companies who own data are the ones exerting their authority over the inside and outside worlds of technology. This is the reason the job of a Data Engineer has become all the more enticing to the new crop of engineers coming out of college. What is a Data Engineer? Def. dumbed down - A person who collects all raw data; transforms and maintains it in databases on computers for task-specific usage. A valuable asset for anyone looking to break into the Data Engineering field is understanding the which are as follows: different types of data → Data is organized into tables with rows and columns. Structured Data → Data in the form of XML, CSV, or JSON files Semi-structured Data → Data from Emails / PDFs Unstructured Data → Data such as audio or image files. Binary Data The level of organization within these types of data follows a top to bottom order with Structured Data being the most organized and Binary Data being the least. Data Pipeline Data Engineers create a that prepares data for the task at hand. Data Pipeline Creating a data pipeline is a three-step process: Acquiring Data Data Ingestion - Ingesting Data into a Data Lake Data Transformation - Saving Data into a Data Warehouse The data in a is in its raw form, while the data in is transformed for a specific purpose. Data Lake Data Warehouse A data scientist would prefer the Data Lake since it has more data. A data analyst would prefer the Data Warehouse as it has data specific to the job. ETL → “Extract, Transform, and Load” is the pipeline used by Data Engineers Types of Databases The different types of Databases are as follows: Relational Databases - Postgres, MySQL NoSQL - MongoDB NewSQL Databases - Vault DB Search Databases - Elastic Search Computation Databases - Apache Spark Data engineering is a more and more enticing career option. Hopefully, anyone looking to break into the Data Engineering field found this story helpful.
