Data Science and Engineering

Retrieval-Augmented Generation (RAG) Introduction Retrieval-Augmented Generation (RAG) is a method/approach that basically enhances LLMs (Large Language Models). It extracts relevant information from external sources such as databases, APIs, Documents etc. before providing any response. Uses of RAG LLMs are limited to training data only and there are high chances that it may provide outdated or incorrect information. Well, worry not, RAG is there for rescue.  RAG solves this by extracting the most recent and relevant data at query time , leading to more accurate, fact-based responses to the user.  RAG improves LLMs by integrating external knowledge retrieval. Let us see how RAG works Retrieve: When a user asks a question, the system fetches relevant documents or data. Augment: The retrieved data is added as context for the LLM. Generate: The LLM then generates a response using both its pre-trained knowledge and the retrieved context. Example: Without RAG: 🤖 The presiden...

We can create a Makefile in Python which helps to automate tasks like running your Python scripts, testing, or building your project.  A Makefile is a file used by the make build automation tool to define a set of tasks that should be executed. Each task is written as a target, and it can depend on other tasks or files. In general, structure of makefile are: Target Dependencies Commands Ok, lets understand this with help of an example. Makefile (has no extensions) # Define variables for system Python and pip for virtual environment SYSTEM_PYTHON = python3 VENV_PYTHON = myvenv/bin/python VENV_PIP = myvenv/bin/pip # Target: venv # Creates a virtual environment in 'myvenv' directory using system Python. venv : $(SYSTEM_PYTHON) -m venv myvenv # Target: install # Installs dependencies if 'requirements.txt' exists. install : venv $(VENV_PIP) install -r requirements.txt; # Target: all # Runs venv, install, and run targets. all : venv install In example above...

 Pyspark Transformation and Actions RDD Transformation Examples Some of the examples of RDD transformations are: flatmap() map() reduceByKey() filter() sortByKey() RDD Actions Example some of the examples of RDD actions are: count() first() max() reduce() take() collect() lets us understand by code how RDD is fault tolerant, immutable, how lazy evaluation works and also understand its distributed nature. lets define a RDD , while converting a data to RDD, it is not directly loaded in memory, first of all  a lineage graph is made for the data, such that even if any node crashes in future, data can be recreated. This lineage graph helps in fault tolerance by allowing Spark to recreate lost data partitions in case of any reason causing node failure. RDDs are distributed across multiple nodes in a cluster, allowing parallel processing of data records. rdd = sc.parallelize([ 1 , 2 , 3 , 4 , 5 ]) the immutability nature of RDD can be understand such that, notice now we didnt changed...

Brief Introduction to Docker img src Docker is a platform as a service (PaaS) for developing, shipping and running applications in containers. By containers, i mean, a lightweight, portable and self-sufficient unit capable of running applications and their dependencies. It is a runnable instance of a Docker image. Docker container runs inside a host machine but completed isolated from host machine. Importance of Docker img src Some of the importance of docker are highlighted as: Dependency Management As data engineers we work on various pipelines and applications. So, to simplify the dependency and make sure we don't end up with conflict across different environment, it is necessary to encapsulate the dependencies of applications in a container and generate an image for it. Portability Docker containers are portable and can be easily moved between different environments, which makes it simpler to deploy and share pipelines and applications across teams or organizations easily. Envi...

Introduction SQL i.e. Structured Query Language is a querying language for relational database. SQL has many flavors some of which are MSSQL, MYSQL, SQLite, PostgreSQL etc. All of these are for manipulation and management of relational databases. SQL provides set of commands that help interact with the database such as creating, modifying and querying data. Like any other languages SQL has levels of learning,  i.e. beginning, intermediate and advance. We will talk about thin brief later on. With SQL we can manage our data in more organized form like in table easy for us to query them when ever we need such as deleting, inserting, update.  Well SQL does have many types and variation of its own and each have their own purpose. They are  Data Declarative Language (DDL) --- Create, Alter, Drop Data Manipulation Language (DML) --- Select, Insert, Update, Delete Data Control Language (DCL) --- Grant, Revoke Transaction Control Language (TCL) --- Commit, Rollback, Savepoint Data...

What is Data? Before jumping to Data Engineering, let's start from what actually is Data. Well in today's world data is everything and everything is data. While scrolling social media, playing games online, surfing internet etc., terabytes of data are being processed and stored. Data nowadays are asset to people, So these assets need to be well managed for taking more benefits as we can. Data Lifecycle If we take a look at impact of data in every gadget and tech we use, we never actually think how the data is reaching us in such form and from where? Data goes through many processes for it to be used in real world some data are used as it is created like in real time system whereas some are used better with certain time intervals. First data is collected from various sources then it is ingested to the pipeline and then it is stored . The data is the processed i.e. converted to the form as desired by the users and used for analysis. The processing storing and analyzing steps are ...