An Introvert's Paradise

Working at the intersection of business and technical teams brings perks and growth opportunities for any data engineer. With this great opportunity comes the responsibility to be a bilingual (if not multilingual) person who can switch the language based on the stakeholder while interacting with them. A data engineer typically interacts with their immediate customers like data analysts, data scientists or machine learning engineers for their different data needs. It is important that we as data engineers not only focus on learning about technical stuff like new features or some bug in spark, but also spend time to understand about the actual business of our clients. Of course, considering people think data engineers are a different version of software engineers (opinionated statement I know), we are expected to be techies! Knowing how the business operates in the context of data will be handy when there is one odd data point, coming from that Kafka source or some third-party data sourc...

If you landed here from LinkedIn, then you probably have the right context! Let's continue on the topic directly:   As per my understanding and experience in #datafield so far, data like a software has lifecycle! 🚲 Few stages: i) Inception (collection of data from disparate systems e.g., transactional systems, sensors, etc.) ii) Collection (the generated data is collected through different methods and stored at a place) iii) Cleaning (generally the data collected isn’t processing ready hence some sort of preparation is required) iv) Processing (using various business logics and/or logical transformations the data is processed so that it gives out some information) v) Presenting (the well-processed data is then presented using various dashboarding tools/techniques e.g., Tableau, Power-BI) vi) Intelligence (using the data, the machine learning models are built which can identify patterns and predict the future patterns using mathematical/statistical methods)   ...