2.4 Using Python

The base functionality of Python is provided in this section. Additional functions and explanations relating to specific methods or algorithms are provided in their respectful chapters in this book. We note that Python 3.6 or higher should be used (Python 2.7 is an older legacy version with which some of the code from this book will not work).

2.4.1 Python setup

There are a number of ways to setup Python on your machine. We will outline the three most frequent methods below:

• The standard Python installation;
• The Miniconda distribution of Python;
• The Anaconda distribution of Python.

Both Miniconda and Anaconda distributions utilise the conda package in their Python installations, which allows to download and install additional Python packages. The standard Python installation uses the pip package to download and install additional Python packages.

2.4.1.1 What’s the difference between pip and conda?

In short, pip allows us to only install Python packages. In other words, we would not have the ability to easily install additional non-Python libraries. In contrast, conda is a packageing tool and installer, which handles library dependencies outside of Python-only packages, as well as the Python packages themselves.

You can use conda and pip side-by-side, however you cannot use them interchangeably - pip cannot install conda format packages.

2.4.1.2 What’s the difference between Miniconda and Anaconda?

The differences are as follows:

• Miniconda = Python + conda
• Anaconda = Miniconda + conda install anaconda

In other words, Anaconda contains an additional (~160) Python packages than the miniconda distribution.

Take note that these additional packages may result in a total installation time of ~40-60 minutes for Anaconda. This also means that if you need to reinstall Anaconda, you will need to wait ~20-30 minutes for the uninstall process to complete, and then an additional 40 - 60 minutes for the installation to complete.

2.4.1.3 Which version should I choose?

For classes, it is recommended to choose the Anaconda distribution, as it contains most of the packages needed.

Alternatively, you can install Miniconda and the appropriate packages, e.g. see the beginning of Ch.3.11, or Ch.4.11. Installing Miniconda should take less time than Anaconda and may be faster, in case you need to reinstall it later.

Finally, only choose to install the standard Python installation, if you have some programming experience and are not afraid of messing with packages installation, which may require configurating additional library dependencies manually.

Only use one method to setup your Python environment, as having more than one installation may cause software conflicts!

2.4.1.4 Installing Python via the Anaconda distribution

Note: the website design for Anaconda has changed, as well as the website itself - www.anaconda.com.

We can install the Anaconda distribution of Python as follows:

Download the appropriate version depending on your operating system:

Make sure you download Anaconda for the latest version of Python:

Again, do not use Python 2.7 as the code syntax and package compatibility will break.

When installing anaconda, make sure that the following boxes are checked (unless you already have an existing non-anaconda python distribution installed):

Finally, after installing anaconda, launch the Anaconda Navigator and go to the packages:

There, check for any updates:

Then, navigate back and update JupyterLab:

After updating JupyterLab, you can update the remaining packages by opening the terminal:

and inputting:

conda update --all

2.4.2 A quick way to launch JupyterLab

On some systems launching the Anaconda navigator may take some time and since we are only interested in JupyterLab, we will make it easier for ourselves by creating an executable for JupyterLab with a custom home directory. Doing so is as straightforward as creating a folder called PrEcon on your desktop:

Then, open up Notepad and input:

cd C:\Users\YOUR_PC_USER\Desktop\PrEcon
jupyter lab

Replace YOUR_PC_USER with your PC user and save the file on your desktop as JupyterLab.bat. In my case this is:

Make sure that you have selected ‘All Files’ for the file type.

One you double click on the .bat file, you will open up a window in your browser but do not close the terminal window as this will close JupyterLab!:

2.4.3 Introductory Python tutorial

Note that most of the functions and methods used in this book will be provided in each chapter. This sections serves only as a quick introduction to the basic functionality of Python.

In general, it is recommended to do either the Introduction to Python tutorials or The Python language from the Scipy Lecture Notes for a quick introduction without any additional software requirements.

On the other hand, similarly to R’s swirl package, we can install PyCharm Edu and get an interactive tutorial (unlike R, here we need to use a different application, instead of an additional package).

2.4.3.1 Python language tutorial using PyCharm

Download PyCharm Edu and install it. Make sure that you already have Anaconda (or alternatively, the base Python but not both as it may cause errors) installed before installing Pycharm Edu.

Once installed, start PyCharm Edu:

If you are certain that everything installed correctly, click Learn to browse courses and select Introduction to Python. Alternatively, to verify that everything works correctly, you can click Create New Project.

You can name your project anything you want and click Create.

In case you have a Python error that python_d.exe is not found when PyCharm creates the Project - see this question on stackoverflow.

Inside the Project select File -> Learn -> Browse Courses:

in the new dialog window select Introduction to Python:

and click Join.

Note:

• The Interpreter:... may be different (for example, conda if you are using Anaconda) - don’t change it unless you know what you are doing - PyCharm Edu selects an available interpreter automatically.
• In case you get a message that the PyCharm interpreter is not configured (even though you have selected a Python version/interpreter) - wait for the Indexing... to finalize and restart PyCharm.

Finally, the selected course will be loaded:

• The left window is the available lessons.
• The middle window is your code and input window - note the highlighted text type your name, where you need to input your name in the first task.
• The right window contains the description of the task, as well as allows you to look at the hints, if you get stuck.

After inputting the required fields, you can click the green arrow to run your code in the script file:

The bottom window will automatically open and show the output of the script.

After examining the output and feeling confident about your answer, click the Check button. You can examine the output of the by clicking on Run in the bottom-left:

If you want to try some other commands and examine their output - you can click on Python console and type some commands in the console at the bottom to execute them one-by-one (as opposed to the script file in the middle window, which executes all of the commands if you press Check or click the previously mentioned green arrow to execute the code).

Finally, click Next to go to the next lesson.

If you accidentally opened more than one tutorial, you can manage your existing projects (open previously saved projects or delete existing ones) via File -> Open Recent -> Manage Projects:

This interactive tutorial will help you familiarize yourself with the basic functionality and syntax of the Python programming language.

After getting familiar with Python iteself, we can move on to JupyterLab, where we will examine hwo we can blend together Python code, its output, add some comments, text formatting as well as mathematical formulas in one document. This makes it easier to have templates/examples of data analysis tasks with model estimation code and result interpretation, without having to spend extra time by copy-pasting them in some other document.

2.4.3.2 Introductory JupyterLab notebook tutorial

Launch JupyterLab and create a new notebook file:

and rename it to python_intro:

There are three different cells to choose from:

• Code - this type of cell treats the input as python (because we created a python notebook) code;
• Markdown - this type of cell treats the input as markdown code;
• Raw - the input is treated as raw text;

You also have a menu to:

• save changes to your notebook;
• add a new cell of the selected type to your notebook;
• cut a selected cell
• copy a selected cell;
• paste the copied cells;
• run a selected cell;
• stop a selected running cell;
• restart the notebook kernel - this clears the current workspace of any variables and loaded packages and is somewhat equivalent to restarting RStudio.

Next, create three different blocks with the following:

• Code cell with:

print("Hello, world!")
x = 2
print(x)

• Markdown cell with:

# This is a title

This is some sample text. **This is some bold text**. 
*This is some text in italics*.

$X = 3$, $Y = 4$

## This is a subtitle

$$
\sum_{k = 0}^{\infty} ar^k = \dfrac{a}{1-r}, \text{ for } |r| < 1
$$

- This is a list item
- This is another list item
    - This is a nested list item
    - This is another nested list item
    
### This is a subsubtitle

1. Some sample text and a formula: $X = Y + Z$
2. Some sample text and a formula: 
  $$ 1 + r + r^2 + r^3 + ... = \lim_{n \rightarrow \infty} \left( 1 + r + r^2 + ... + r^n \right)$$
    1. Some sample text
    2. Some sample text
3. A matrix: $\begin{pmatrix}
a & b \\ 
c & d 
\end{pmatrix} =
\begin{pmatrix}
a & b \\
c & d
\end{pmatrix}
$

Another, centered matrix:
$$
\begin{align}
\begin{pmatrix}
a & b \\
c & d
\end{pmatrix} &=
\begin{pmatrix}
a & b \\
c & d
\end{pmatrix} 
+ 
\begin{pmatrix}
a & b \\
c & d
\end{pmatrix}\\
&-
\begin{pmatrix}
a & b \\
c & d
\end{pmatrix} 
\end{align}
$$

• Raw cell with:

print("Hello, world!")

- This is a list item
- This is another list item
    - This is a nested list item
    - This is another nested list item

x = 3
print(x)

$X = Y + Z$

You can either compile a selected cell by pressing CTRL + ENTER, or all the cells with:

Notice that the Raw cell doesn’t produce any output and doesn’t compile any LaTeX / Markdown code.

2.4.3.3 Python programming language at a glance

Below we present some code examples of Pythons code syntax. Explanations are minimal - the idea is to have quick examples with output to verify how Python works. For more in depth examples, see the previous subsection.

2.4.3.3.1 Strings

Run the following code and verify that you understand what happened to the output:

Assign, print and transform strings:

x = "this is a sentence"
print(x)

## this is a sentence

print(x.capitalize())

## This is a sentence

print(x.title())

## This Is A Sentence

print(x.upper())

## THIS IS A SENTENCE

Split a string into a list of words and select different elements from the list:

y = x.split(" ")
print(y)

## ['this', 'is', 'a', 'sentence']

print(y[0])

## this

print(y[0:1])

## ['this']

print(y[0:2])

## ['this', 'is']

print(y[0:3])

## ['this', 'is', 'a']

print(y[-1])

## sentence

print(y[-2:])

## ['a', 'sentence']

print(y[0:2])

## ['this', 'is']

print(x.split(" ")[:-1])

## ['this', 'is', 'a']

print(x.split(" ")[:-2])

## ['this', 'is']

Combine different strings:

print(" ".join(x.split(" ")[:-1]))

## this is a

print("A" + "B")

## AB

y = "dog"
print(" ".join(x.split(" ")[:-1]) + " " + y)

## this is a dog

z = " ".join(x.split(" ")[:-1]) + " " + y.upper()
print(z)

## this is a DOG

print((z + ".").capitalize())

## This is a dog.

Trim white-space, add line breaks and tab spacing:

a = "     a     "
print("'" + a + "'")

## '     a     '

print("'" + a.lstrip() + "'")

## 'a     '

print("'" + a.rstrip() + "'")

## '     a'

print("'" + a.strip()  + "'")

## 'a'

print("\t" + x)

##  this is a sentence

print("\n" + x)

## 
## this is a sentence

print("----" + "\n\t\t" + x + "\n----")

## ----
##      this is a sentence
## ----

2.4.3.3.2 Numbers

Run the following code and verify that you understand what happened to the output:

Assign values to variables, print the values with a string text and perform basic math operations:

x = 2
y = 3
print("x = " + str(x) + " y = " + str(y))

## x = 2 y = 3

print(x + y)

## 5

print(x - y)

## -1

print(x * y)

## 6

print(x+y*y)

## 11

print((x+y)*y)

## 15

Carry a value to the power of different values:

print(x**0)

## 1

print(x**1)

## 2

print(x**2)

## 4

print(y**x)

## 9

print(x**y)

## 8

print(25**0.5)

## 5.0

2.4.3.3.3 Lists

A list can store multiple variables. The variables need not be of the same type.

Print different items in a list, combine different lists, etc.:

x = ["dog", 11, "cat", 13.5]
y = ["car", 25**0.5, "pencil", 0.5+3]
print(x)

## ['dog', 11, 'cat', 13.5]

print(x[0])

## dog

print(x[1])

## 11

print(x*0)

## []

print(x*2)

## ['dog', 11, 'cat', 13.5, 'dog', 11, 'cat', 13.5]

print(x + y)

## ['dog', 11, 'cat', 13.5, 'car', 5.0, 'pencil', 3.5]

print(x[0].upper())

## DOG

Note the different data types:

print(type(x))

## <class 'list'>

print(type(x[0:1]))

## <class 'list'>

print(type(x[0]))

## <class 'str'>

print(type(x[1]))

## <class 'int'>

print(type(str(x[1])))

## <class 'str'>

Change items in the list:

x = ["dog", 11, "cat", 13.5]
print(x)

## ['dog', 11, 'cat', 13.5]

x[0] = "car"
x[1] = str(x[1])
x[3] = str(x[3])
print(x)

## ['car', '11', 'cat', '13.5']

Add or remove items from a list:

x = []
print(x)

## []

x.append("dog")
x.append("cat")
x.append(x[0] + " & " + x[1])
print(x)

## ['dog', 'cat', 'dog & cat']

x.insert(1, "cow")
print(x)

## ['dog', 'cow', 'cat', 'dog & cat']

x.remove("dog")
print(x)

## ['cow', 'cat', 'dog & cat']

del x[-1]
print(x)

## ['cow', 'cat']

print(x.pop())

## cat

print(x)

## ['cow']

x = ["car"] + x
print(x)

## ['car', 'cow']

x.pop(0)

## 'car'

print(x)

## ['cow']

Finding elements in a list:

x.append("cat")
print(x)

## ['cow', 'cat']

print(x.index("cow"))

## 0

print(x.index("cat"))

## 1

We get an error if we try to print an index of an item which is not in the list:

print(x.index("dog"))

## Error in py_call_impl(callable, dots$args, dots$keywords): ValueError: 'dog' is not in list
## 
## Detailed traceback: 
##   File "<string>", line 1, in <module>

List with numeric values:

numbers = [2, 10, 5, 8, 4, -2, 6]
print("Minimum: " + str(min(numbers)))

## Minimum: -2

print("Maximum: " + str(max(numbers)))

## Maximum: 10

print("Total: " + str(sum(numbers)))

## Total: 33

Sort a list and print its length:

x = ['dog', 'cow', 'cat', 'dog & cat']
print(sorted(x))

## ['cat', 'cow', 'dog', 'dog & cat']

print(x)

## ['dog', 'cow', 'cat', 'dog & cat']

x.sort()
print(x)

## ['cat', 'cow', 'dog', 'dog & cat']

x.reverse()
print(x)

## ['dog & cat', 'dog', 'cow', 'cat']

print("This list contains " + str(len(x)) + " elements")

## This list contains 4 elements

Split strings:

x = "This is a sentence."
print(x)

## This is a sentence.

print(x.split(" "))

## ['This', 'is', 'a', 'sentence.']

print(x.split("a"))

## ['This is ', ' sentence.']

print(type(x))

## <class 'str'>

print(len(x))

## 19

print(type(x[0]))

## <class 'str'>

print(x[0])

## T

print(x[0:6])

## This i

Note that some of the functions, like insert(), remove(), sort(), pop(), etc. change the original elements in x. This is because lists are so called mutable objects. Mutable objects can be changed after they are created. Mutable objects are passed by object reference, instead of value.

This means, that modifying the original value changes any other variable that the value is passed as reference:

x = [1, 2, 3]
y = x
print(x)

## [1, 2, 3]

print(y)

## [1, 2, 3]

x.append(-1)
print(x)

## [1, 2, 3, -1]

print(y)

## [1, 2, 3, -1]

y.append(10)
print(x)

## [1, 2, 3, -1, 10]

print(y)

## [1, 2, 3, -1, 10]

x[0] = x[0] + 10
print(x)

## [11, 2, 3, -1, 10]

print(y)

## [11, 2, 3, -1, 10]

A workaround is to explicitly create a new variable, instead of a reference:

x = [1, 2, 3]
y = list(x)
z = y[:]
print(x)

## [1, 2, 3]

print(y)

## [1, 2, 3]

print(z)

## [1, 2, 3]

x.append(-1)
print(x)

## [1, 2, 3, -1]

print(y)

## [1, 2, 3]

print(z)

## [1, 2, 3]

y.append(10)
print(x)

## [1, 2, 3, -1]

print(y)

## [1, 2, 3, 10]

print(z)

## [1, 2, 3]

z[0] = z[0] + 10
print(x)

## [1, 2, 3, -1]

print(y)

## [1, 2, 3, 10]

print(z)

## [11, 2, 3]

2.4.3.3.4 If Statements

We use if statements to test for some kind of condition.

things = ["milk", "bread", "tomatoe"]
my_search = "milk"
print(my_search == things[0])

## True

print(my_search == things)

## False

print(my_search in things)

## True

my_search = my_search.upper()
print("Changed: " + my_search.lower() + " to: " + my_search)

## Changed: milk to: MILK

if my_search in things:
  print("Found " + my_search)
else:
  print("No " + my_search + " in list")

## No MILK in list

if not my_search in things:
  print("No " + my_search + " in list")
else:
  print("Found " + my_search)

## No MILK in list

if len(things) >=4:
  print("There are at least 4 elements in the list")
elif len(things) ==3:
  print("There are 3 elements in the list")
else:
  print("There are less than 3 elements in the list")

## There are 3 elements in the list

numbers = [1, 2, 3, 4]
print(1 in numbers)

## True

print(6 in numbers)

## False

print("1" in numbers)

## False

print("1" in str(numbers))

## True

print(1 == 2)

## False

print(1 != 2)

## True

print(1 in numbers or 6 in numbers)

## True

print(1 in numbers and 6 in numbers)

## False

print(1 in numbers or 2 in numbers)

## True

print(1 in numbers and 2 in numbers)

## True

2.4.3.3.5 Loops

We can loop through each item in a list. Note that we need to transform any non-strings to strings if we want to print and concatenate the value into a string:

item_list = ["car", 11, "pencil", 13.5]
for i in item_list:
  print("Item in the list: " + str(i))
  

## Item in the list: car
## Item in the list: 11
## Item in the list: pencil
## Item in the list: 13.5

for i in range(0, len(item_list)):
  print("Item " + str(i) + " is: " + str(item_list[i]))

## Item 0 is: car
## Item 1 is: 11
## Item 2 is: pencil
## Item 3 is: 13.5

Format a list as a numbered list via enumerate():

list_format_1 = ")"
list_format_2 = " ;"
for index, value in enumerate(item_list, 1):
  print(("{}" + list_format_1 + " " + "{}" + list_format_2).format(index, value))

## 1) car ;
## 2) 11 ;
## 3) pencil ;
## 4) 13.5 ;

In the above example our numbered list started at 1. The list index numbers and the list values are printed in the {} symbols. Each list number is formated as i), followed by the list element value and with the ; symbol appended to the end. If we wanted, we could change, or remove these extra formatting options.

We can also create the formatting in a different way:

item_list = ["This", "is", "a", "list"]
for index, item_i in enumerate(item_list):
    print("Item index is: " + str(index) + ", item #" + str(index + 1) + 
          ", item: " + item_i.title())

## Item index is: 0, item #1, item: This
## Item index is: 1, item #2, item: Is
## Item index is: 2, item #3, item: A
## Item index is: 3, item #4, item: List

range() function:

for number in range(1, 5):
    print(number)

## 1
## 2
## 3
## 4

numbers = list(range(1, 5))
print(numbers)

## [1, 2, 3, 4]

2.4.3.3.6 Tuples

Tuples are sequences, just like lists. The differences between tuples and lists - tuples cannot be changed, unlike lists, and tuples use parentheses, whereas lists use square brackets. Tuples are immutable which means you cannot update or change the values of tuple elements. You can, however, take portions of existing tuple variables and create new tuple variables.

item_t = ("This", "is", "a", "tuple")
for index, item_i in enumerate(item_t):
    print("Item index is: " + str(index) + ", item #" + str(index + 1) + 
          ", item: " + item_i.title())

## Item index is: 0, item #1, item: This
## Item index is: 1, item #2, item: Is
## Item index is: 2, item #3, item: A
## Item index is: 3, item #4, item: Tuple

number = 1
print("Value saved (equal to %d)." % number)

## Value saved (equal to 1).

print("Value saved (equal to " + str(number) + ").")          

## Value saved (equal to 1).

color = "green"
print("The color is %s." % color)

## The color is green.

print("My #%d color is %s" % (number, color))

## My #1 color is green

my_tuple = (number, color)
print("My #%d color is %s" % my_tuple)

## My #1 color is green

numbers = [7, 23, 42]
print("The numbers are %d, %d, and %d." % (numbers[0], numbers[1], numbers[2]))

## The numbers are 7, 23, and 42.

2.4.3.3.7 Dictionaries

Dictionaries allows storing data in key-value pairs.

import numpy as np
my_dictionary = {"name": "Python",
                 "is_active": True,
                 "test_scores": [9, 8, 10]}

if my_dictionary["is_active"]:
  print("Active language is " + my_dictionary["name"] +
        " with average score of %f" % np.mean(my_dictionary["test_scores"]))

## Active language is Python with average score of 9.000000

my_dict = {'key_1': 'value_1', 'key_2': 'value_2', 'key_3': 'value_3'}

for key in my_dict:
    print('Key: %s' % key)
    

## Key: key_1
## Key: key_2
## Key: key_3

for key, value in my_dict.items():
    print('Key: %s and value: %s' % (key, value))   

## Key: key_1 and value: value_1
## Key: key_2 and value: value_2
## Key: key_3 and value: value_3

%d is the format code for an integer, %f is the format code for a float.

my_dictionary["test_scores"] = [9, 8, 5]
print("%f" % np.mean(my_dictionary["test_scores"]))

## 7.333333

print("%d" % np.mean(my_dictionary["test_scores"]))

## 7

my_dictionary["test_scores"] = [9, 8, 9]
print("%f" % np.mean(my_dictionary["test_scores"]))

## 8.666667

print("%d" % np.mean(my_dictionary["test_scores"]))

## 8

print(my_dictionary.keys())

## dict_keys(['name', 'is_active', 'test_scores'])

print(my_dictionary.values())

## dict_values(['Python', True, [9, 8, 9]])

2.4.3.3.8 Functions

#Define a function
def my_sum_function(x1, x2):
  #Add function code
  print("Adding values ...")
  return x1 + x2

Use The function:

x1 = 1
x2 = 2
print(my_sum_function(x1, x2))

## Adding values ...
## 3

Define another function:

def number_to_word(number):
  if number == 0:
    return "zero"
  elif number == 1:
    return "one"
  elif number == 2:
    return "two"
  elif number < 0:
    return "negative"
  else:
    return "greater than two"

Use the function in a loop:

numbers = [0, 1, -1, 3]
for i in range(0, len(numbers)):
  print("The number is " + number_to_word(numbers[i]))

## The number is zero
## The number is one
## The number is negative
## The number is greater than two

2.4.3.3.9 Classes

Classes allow combining information and behaviour. For an example, see 2.7.7.