dbt - Common Setup Issues behind Corp Security (Windows)

Hi,

In this article, I would like to share and archive my experience facing common issues working with DBT for data transformation projects implemented behind corporate security setup.

Starting with a windows laptop behind the modern-day corporate firewall/security solutions like ZScaler/Netskop where SSL Inspection is in place.

The following errors and workable solutions listed below; are the ones I face commonly:

Error: PIP Install (SSL: CERTIFICATE_VERIFY_FAILED)

Databricks setup | dbt Developer Hub (getdbt.com)

pip install dbt-databricks        

Solution:

For this error, one has to amend their pip config [pip.ini] file in order to make it work.

So how do you find where your config file is located:

Note: if all the location suggested above are False, then navigate to the Global location and create a pip.ini file manually by oneself with the content of the file listed below.

[global]
trusted-host = pypi.python.org
               pypi.org
               files.pythonhosted.org        

And the error is resolved!!

Error: DBT deps (SSL: CERTIFICATE_VERIFY_FAILED)

As soon we have initialized our DBT project with all the outputs of dbt debug OK. and head over to install the complimenting packages of DBT to make our work more improved via dbt deps, and we stumble on next error:

dbt deps
23:17:38  Running with dbt=1.7.10
23:17:44  Encountered an error:
External connection exception occurred: HTTPSConnectionPool(host='hub.getdbt.com', port=443): Max retries exceeded with url: /api/v1/index.json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)')))        

Upon investigating this error further, I seem that this error is specific to window OS version and not prevalent in Mac/Linux distributions.

Solution:

Install the following package and error is gone!!

python-certifi-win32 · PyPI

pip install python-certifi-win32        

I hope the above solutions would benefit any one in a similar situation faced.