Quick Tip: The headache caused by import statements in Python

When developing applications, there has to be a method to the madness.

Just because a programming environment allows certain elements, we should use them with a thought.

Databricks provides a very helpful, interactive cell-based development mechanism. In this mechanism, we can write code in a cell, execute it, correct it and execute again. After successful execution of the cell, we go to a new cell and add required code. All is well till we execute things in the proper sequence.

As it happens, we come back to the notebook the next day and jump directly to the cell that we were working on the previous day. When we run this cell, we get package errors. Hence we add the import statements for the packages and go merrily on our way.

And this is where more problems crop up.

When we run the notebook once again or integrate this notebook with other functionality, Such cell level import statements create an issue because things are not imported in the proper order. Nor are some of the packages imported correctly.

To overcome this issue, it is important that we import packages only in one cell.

A better practice is to import the package using a unique name, removing the issue of incorrect resolution.

A package that commonly causes issue in Python is datetime. Due to our infinite wisdom, the newer datetime is defined inside datetime. Logically, it should have been name datetime2. But this name sounds too crappy and hence not used. So we find statement like this

from datatime import *

and this as well

from datatime import datetime

In such cases, if we use datetime.now(), depending on the import statement used, we will get an error. If we use style 1, we have to use datetime.datetime.now(). If we use style 2, we have to use datetime.now(). This causes a lot of confusion and maintenance headaches

LESSONS TO LEARN

  1. import in consistent manner
  2. place imports in one cell at the top of the notebook
  3. provide unique name for the import


#python #import #quick_tip

要查看或添加评论,请登录

Bipin Patwardhan的更多文章

  • Change management is crucial (Databricks version)

    Change management is crucial (Databricks version)

    My last project was a data platform implemented using Databricks. As is standard in a data project, we were ingesting…

  • Friday fun - Impersonation (in a good way)

    Friday fun - Impersonation (in a good way)

    All of us know that impersonation - the assumption of another person's identity, be it for good or bad - is not a good…

  • Any design is a trade-off

    Any design is a trade-off

    Irrespective of any area in the world (software or otherwise), every design is a trade off. A design cannot be the 'one…

    1 条评论
  • Databricks: Enabling safety in utility jobs

    Databricks: Enabling safety in utility jobs

    I am working on a project where we are using Databricks on the WAS platform. It is a standard data engineering project…

  • A Simple Code Generator Using a Cool Python Feature

    A Simple Code Generator Using a Cool Python Feature

    For a project that I executed about three years ago, I wrote a couple of code generators - three variants of a…

  • Recap of my articles from 2024

    Recap of my articles from 2024

    As we are nearing the end of 2024, I take this opportunity to post a recap of the year - in terms of the articles I…

  • Handling dates

    Handling dates

    Handling dates is tough in real life. Date handling is probably tougher in the data engineering world.

  • pfff -- why are you spending time to save 16sec execution time

    pfff -- why are you spending time to save 16sec execution time

    In my current project, we are implementing a data processing and reporting application using Databricks. All the code…

    2 条评论
  • Quick Tip - Add a column to a table (Databricks)

    Quick Tip - Add a column to a table (Databricks)

    As the saying goes, change is the only constant, even in the data space. As we design tables for our data engineering…

  • Friday Fun - Reduce time of execution and face execution failure

    Friday Fun - Reduce time of execution and face execution failure

    In my project that has been executing since Dec 2023, things have been going good. We do have the occasional hiccup…

社区洞察

其他会员也浏览了