Logo

dev-resources.site

for different kinds of informations.

Important Questions related to Data Engineering

Published at
3/20/2023
Categories
data
dataengineering
pipelines
questions
Author
zapeeosheikh
Author
12 person written this
zapeeosheikh
open
Important Questions related to Data Engineering

Can a database be used as DWH?

Yes, Database can be used as a Data Warehouse. Even before the data warehouse, we use database for the business. But now it's a good practice to use Data warehouse because of quick response and Analystical tools. Data warehouse is based on specific category thatswhy, we get quick response as compared to Database.

Difference between structured and UN structured data ?

Structured Data:
Is in the form of structured manner, it can be a SQL query or a script that we run on terminal of any platform to perform task is considered as structured data. This data is in uniform format.

Unstructured Data:
This data is in distributed manner. It is in scattered formed. This data contains text file, video, image, audio etc.

Roles of Data Engineer :

  1. Design and develop databases, data models and tables to store data.
  2. Develop ETL processes and data pipelines for data integration.
  3. Monitor and optimize database performance.
  4. Analyze data for patterns, trends, and insights.
  5. Develop data visualizations and dashboards.
  6. Troubleshoot data-related issues.
  7. Implement security and privacy measures.
  8. Collaborate with data scientists and business stakeholders.
  9. Build API connection to fetch the data from the source target to the destination target.
pipelines Article's
30 articles in total
Favicon
The Art of Iteration: Starting the Cycle
Favicon
The Art of Iteration: Loop in Pipeline Stage
Favicon
Automating Docker Workflows with Jenkins: A Complete Guide
Favicon
TIL how to see the entire commit column on GitLab using JS
Favicon
Getting Started with Apache Kafka: A Backend Engineer's Perspective
Favicon
Building pipelines with IAsyncEnumerable in .NET
Favicon
DevOps Security Integrating Best Practices into Your Pipeline
Favicon
Creating a data pipeline using Dataproc workflow templates and cloud Schedule
Favicon
☸️ Kubernetes: A Convenient Variable Substitution Mechanism for Kustomize
Favicon
Setting Up a CI/CD Pipeline with AWS and Git: A Comprehensive Guide
Favicon
Enabling Pipelines: Easier than ever
Favicon
Optimizing GitLab CI for Readability and Maintainability: From 1K to 600 Lines!
Favicon
Building Robust Data Pipelines: A Comprehensive Guide
Favicon
Azure DevOps Pipelines breaks my "additional arguments" when using Deploy to Azure
Favicon
What is CI/CD Pipeline?-Comparing pipelines!
Favicon
🌟 The Power of Automation: Deploying an ARM Template in Microsoft Azure πŸš€
Favicon
Meet cici-tools, a multi-tool for building GitLab CI/CD pipelines
Favicon
Unlocking the Power of Data: 7 Key Factors to Consider When Building Data Pipelines
Favicon
Optimize Development with Jenkins Pipelines and Continuous Integration
Favicon
Amplify Your Tech Stack with Jenkins Shared Libraries
Favicon
Important Questions related to Data Engineering
Favicon
How To Secure Your CI/CD Pipeline
Favicon
Flexible and dynamic flow control of Azure DevOps YAML Pipelines using variables
Favicon
Go API Project Set-Up
Favicon
Sftp with Az Devops
Favicon
Error: Full scoped PAT is restricted by your organisation
Favicon
Error: No hosted parallelism has been purchased or granted
Favicon
SparrowCI - DSL is dead, long live DSL!
Favicon
Introducing the CircleCI Config SDK
Favicon
Run DB Scripts to Azure PostgreSQL Single Server using Azure CLI Task in pipeline

Featured ones: