Skip to main content

Make Automation


We can create a Makefile in Python which helps to automate tasks like running your Python scripts, testing, or building your project. 
A Makefile is a file used by the make build automation tool to define a set of tasks that should be executed. Each task is written as a target, and it can depend on other tasks or files.

In general, structure of makefile are:

  • Target

  • Dependencies

  • Commands

Ok, lets understand this with help of an example.


Makefile(has no extensions)
# Define variables for system Python and pip for virtual environment
SYSTEM_PYTHON = python3
VENV_PYTHON = myvenv/bin/python
VENV_PIP = myvenv/bin/pip

# Target: venv
# Creates a virtual environment in 'myvenv' directory using system Python.
venv:
$(SYSTEM_PYTHON) -m venv myvenv

# Target: install
# Installs dependencies if 'requirements.txt' exists.
install: venv
$(VENV_PIP) install -r requirements.txt;


# Target: all
# Runs venv, install, and run targets.
all: venv install



In example above,

  • install is a Target.

  • Dependencies  is  requirements.txt. 

  • Command  is $(PIP) install -r requirements.txt

Now to run the make file , just type

 ---- command-----
 make all 

in the terminal, as shown in screenshot below.












This is a simple make file that automates the making of virtual environment, then inside that environment , we install pandas.



Comments

Popular posts from this blog

Docker With PostgreSQL

Brief Introduction to Docker img src Docker is a platform as a service (PaaS) for developing, shipping and running applications in containers. By containers, i mean, a lightweight, portable and self-sufficient unit capable of running applications and their dependencies. It is a runnable instance of a Docker image. Docker container runs inside a host machine but completed isolated from host machine. Importance of Docker img src Some of the importance of docker are highlighted as: Dependency Management As data engineers we work on various pipelines and applications. So, to simplify the dependency and make sure we don't end up with conflict across different environment, it is necessary to encapsulate the dependencies of applications in a container and generate an image for it. Portability Docker containers are portable and can be easily moved between different environments, which makes it simpler to deploy and share pipelines and applications across teams or organizations easily. Envi...

Pyspark Transformation and Actions

 Pyspark Transformation and Actions RDD Transformation Examples Some of the examples of RDD transformations are: flatmap() map() reduceByKey() filter() sortByKey() RDD Actions Example some of the examples of RDD actions are: count() first() max() reduce() take() collect() lets us understand by code how RDD is fault tolerant, immutable, how lazy evaluation works and also understand its distributed nature. lets define a RDD , while converting a data to RDD, it is not directly loaded in memory, first of all  a lineage graph is made for the data, such that even if any node crashes in future, data can be recreated. This lineage graph helps in fault tolerance by allowing Spark to recreate lost data partitions in case of any reason causing node failure. RDDs are distributed across multiple nodes in a cluster, allowing parallel processing of data records. rdd = sc.parallelize([ 1 , 2 , 3 , 4 , 5 ]) the immutability nature of RDD can be understand such that, notice now we didnt changed...