1.1 Regarding Some of The Code in This Book

We will suppress deprecation warnings of future package changes when importing packages (we will also suppress runtime warnings, though they should only be suppressed in cases when it is absolutely sure why they occur) with:

dir()

## ['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'r', 'sys']

import warnings
warnings.filterwarnings("ignore", category = FutureWarning)
# Suppressed for code in this book only:
warnings.filterwarnings("ignore", category = RuntimeWarning)
#
import matplotlib.cbook
warnings.filterwarnings("ignore", category = matplotlib.cbook.MatplotlibDeprecationWarning)
#warnings.filterwarnings("ignore", category = UserWarning, module = 'matplotlib.figure')
# https://github.com/ipython/ipython/issues/11167#issuecomment-394115027
#
import sklearn.exceptions
warnings.filterwarnings("ignore", category = sklearn.exceptions.UndefinedMetricWarning)
#
import statsmodels.api as sm
warnings.filterwarnings("ignore", category = sm.tools.sm_exceptions.ValueWarning)
warnings.filterwarnings("ignore", category = sm.tools.sm_exceptions.HessianInversionWarning)
warnings.filterwarnings("ignore", category = sm.tools.sm_exceptions.ConvergenceWarning)

We will also need to sometimes reset the working environment between chapters and remove the existing plot instances:

import matplotlib.pyplot as plt
#
def close_plots():
  my_plots = plt.get_fignums()
  for j in my_plots:
    plt.close(plt.figure(j))
#
import gc
#
def clear_all(clean_env_var = ['R', 
          '__annotations__', '__builtins__', 
          '__doc__', '__loader__', '__name__', 
          '__package__', '__spec__', 'r', 'sys',
          'warnings', 'gc']):
    # Clears all the variables from the workspace
    gl = globals().copy()
    clean_env_var.append(['clean_env_var', 'gl'])
    #print(len(gl))
    for var in gl:
        if var in clean_env_var: continue
        del globals()[var]
    # Garbage collection:
    gc.collect()
#
#After each script run the following to clear the environment:   
# close_plots()
# clear_all()

For R code this will be done automatically by running the following lines:

#Detach all non-base packages:
detachAllPackages <- function() {
  #Get all other packages:
  pkgs = names(sessionInfo()$otherPkgs)
  #Remove the attached packages:
  if(length(pkgs) >0){
    pkgs = paste('package:', pkgs, sep = "")
    for(package in pkgs){
      detach(package, character.only=TRUE)
    }
  } 
}
detachAllPackages()

#Clean the workspace:
rm(list = ls(all = TRUE))

This is a compatibility issue when writing this book to compile both R and Python code at the same time. The functions clear_all() and close_plots() should be omitted in your own code as they are only specifically required for compilation of this book. The reason for compiling the code along with the book format is that it ensures that all of the functionality and output is unchanged with the given code samples and library versions.

An example of how the Python environment is cleared is provided below. Firstly, the variables in the environment are:

print(dir())

## ['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'clean_env_var', 'clear_all', 'close_plots', 'gc', 'matplotlib', 'plt', 'r', 'sklearn', 'sm', 'sys', 'warnings']

The above depends on your specific Python environment. For example, Spyder IDE has additional global variables, which should not be removed!

Next, we will create some example variables, and load some packages:

import numpy as np
import statsmodels.api as sm
import scipy.stats as stats
import scipy.optimize as optimize
import pandas as pd
#
VARIABLE_X = list(range(0, 10))
x1 = 1
x2 = 2
x3 = [1, 2, 3]
y = "Text"

Now, the working environment has the following variables:

print(dir())

## ['VARIABLE_X', '__annotations__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'clean_env_var', 'clear_all', 'close_plots', 'gc', 'matplotlib', 'np', 'optimize', 'pd', 'plt', 'r', 'sklearn', 'sm', 'stats', 'sys', 'warnings', 'x1', 'x2', 'x3', 'y']

Next, we clean the environment:

clear_all(clean_env_var = ['R', '__annotations__', '__builtins__', 
          '__doc__', '__loader__', '__name__', '__package__', '__spec__', 
          'r', 'sys', 'warnings', 'gc'])

and see that the remaining variables are the ones from the initial environment:

print(dir())

## ['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'gc', 'r', 'sys', 'warnings']