Intermediate Python Refresher: Tutorial, Project Ideas, and Tips

This article is to teach Python beginners and developers some key concepts used in Python that aren't taught from the get-go.

Last updated: May 17th, 2020

If you can create a quadratics root solver, you'll be able to understand this article. These are some concepts I didn't learn in one day but rather a couple years as I am self-taught.

I tried to make this into a video, but the video ended up being 1:45
hours long; I myself would need a lot of motivation to watch a
“tutorial” that long, and I prefer articles to get my information
since I can pick out relevant details rather than trying to skip
sections of a video.

How It All Started

I learned Python basics through CS Circles, and then proceeded to improve/test my problem solving skills. I did this by doing CCC questions which you can find (among other contest problems) at DMOJ. Other sites to improve your algorithmic problem solving skills include HackerRank and LeetCode (although this is mainly for interview preparations).

While I was doing this, I was coding with the default IDLE editor! I did this for 3 months and then I found out about PyCharm which
has a slight learning curve but is loads better in terms of features
and at improving productivity. Nowadays, I use both PyCharm and Visual Studio Code.

I personally have an entire folder dedicated to snippets of code I could
use in the future, so I suggest you also do that and maybe even add some
of the code snippets in this article in there so that you can read your
own examples instead of Googling them or coming back to this article.

General Tips

These are some tips that are not bound to programming but just life and productivity in general.

Know your keyboard shortcuts
Know both the program specific ones (browser, explorer, IDE of choice, etc.) and also OS specific ones (e.g. Win + R for run).

Know the command line

Instead of doing a calculation by hand or opening an IDE to create and run a script, you can actually execute Python code from the command line.
Aside from the common batch functions (e.g. ls, cd), knowing how to use
Python from the command line will save you a lot of time

Command Line Python Example

Know how to Google

Google (or your search engine of choice), is my best friend and should also be yours. It has saved me a lot of time and so it could also save you a
lot of time. It can’t do that if you don’t use it or don’t know how to
use it. When you Google something, your query needs to be general enough that you can find answers, but also specific enough so that those
answers are relevant.

Problem Breakdown Strategy

This goes hand in hand with Googling. Suppose you have a problem/project. You need to break it down into smaller parts. You then need to analyze each of these parts and see if they are small enough for you to complete each of them. If not, either your missing some knowledge that you should Google or the part is too big and needs to be broken down again. You keep doing this recursive procedure until your project has been
split into solvable parts so that you can complete them and then weave
together a project. When I search and find answers through Google, I
don’t expect them to be 100% what I need. I usually need to remix
them into what I want and that’s what you should also expect: the bare
minimum solution that takes you at least one step forward.

With these tips stated, you can do a couple of different things next. You
can skim the rest of the document and make notes on the snippets of code
I feature (what I would do personally), read only the headings, skip to
the project ideas section, or stop reading altogether as my tips are so
useful.

Refresher

In CS Circles, they bring abou`t the print function and some of its
optional parameters but it’s easy to forget about them so here it is
again.

>>> # The default parameters for print are sep=' ', and end='\n'
>>> print('4', '21', 2020, sep='/', end='\n---------\n')
4/21/2020
---------

Let’s Start

Input function and String Formatting

The input function has an optional parameter so that it can also act as a
prompt and if you are using Python 3.6+, you can make use of f-strings.

name = input('Enter your name: ')
print(f'Hello {name}!')  # modern way of string formatting
# if input='reader', output: Hello reader!

For Loops

I want to make clear to you that a for loop, is not a while loop as it is
in other languages. In Python, a for loop is an iteration over an iterable object.

The range function has three parameters, two of them being optional. Do not write use the range function with an explicit start value of 0 because 0 is the default start value, (unless of course you are modifying the default step value of 1).

In this example, I will show you exactly what I mean by “not a while loop” and how a for loop (specifically range) does not add to the temporary value.

# range(start=0, stop, step=1)
# range(5) == range(0, 5) == range(0, 5, 1)

for i in range(5):
    print(i)
    i += 2
# Guess the output. HINT: i += 2does nothing

If you run this code, you’ll notice that the output is increasing by 1
each time even if we are adding 2 to i at every loop. This is because i is
set to the next value in range and isn’t actually being increased by
one each time. This means that we can actually iterate over all sorts of
iterable objects, like lists, without having to use range and indexing.

some_letters = ['a', 'b', 'c', 'd', 'e']
for letter in some_letters:
    # do something
    pass

Here I introduced the keyword pass, this is to avoid errors in otherwise empty blocks.

If you do want to keep track of the index as well as the item, you still
don’t have to use range, you can use the built-in function enumerate.

for i, letter in enumerate(some_letters, start=0):
    print(f'item at index {i} is {letter}')

You can think of enumerate as turning an iterable into an iterable of pairs (index, item of iterable at index).

You can also use the next function to retrieve the next value in an
iterable (if there is no next item, an error will be raised).

File I/O

I’m including this because if you search “how to read files in python” on Google, you are given this which teaches it the old way and not the modern approach.

# make sure test.txt exists with text in it

# OLD 
f = open('test.txt')  # note default is mode='r'
# do something with f here
f.close()
with open('test.txt') as f: # NEW; no close() needed
   print(f.read())
   # f.read() moves the "cursor" to the end of the file
   assert not f.read()
   f.seek(0)
   assert f.read()
   # f.read() returns a string now (unless test.txt is empty)

with open('test.txt', 'w') as f:
    # f.read()  ERROR do not do this
    f.write('this is a test\n')  # note there is no end parameter
    f.writelines(['line1\n', 'line2\n'])  # note no auto newline# other modes: a for append, rb for reading-bytes, wb for writing bytes, and r+/w+ for both reading and writing at the same time

If you are curious why the r+/w+ is not the default, think about how a file
cannot be opened to be written to if it is being “written” to by
another program. If you just need to read a file, it being open in
another program means that you won’t be interfering with the other
program.

Error Handling

# handling the error
try:
    raise RuntimeWarning('Something could go wrong')
except RuntimeWarning as e:  # as e is optional
    # handle the exception here

# ignoring the error
# old
try:
    raise Exception('BOO')
except Exception: pass

# new
from contextlib import suppress
def ignore_error(exception: Exception): 
    """
    Use three quotes for docstrings or long strings
    """
    # : for type hinting (in a dynamic typed language!) and
    # yes you can pass exceptions and functions as parameters
    with suppress(exception):
        raise exception('BOO')
        print('not printed')

ignore_error(RuntimeError)
print('this gets printed')

By this point if you are following along in PyCharm, you would have seen some squiggly lines, especially under “Exception” in the above code.
These squiggly lines help you to avoid syntax errors, follow style
guidelines, and bring attention to code that could be doing something
you didn’t want it to be doing.

More Data Types

So what are these iterables I keep mentioning? Yes a list is an iterable and so are tuples (which you should already know of).

There are also dictionaries, sets and generators (not discussed here).
Dictionaries are like hash tables in other languages, because they
“hash” the key to store information.

empty_dict = {}  # or dict()
my_dict = {'key': 'value'}
# How to get value from dict
my_dict['a']  # raises KeyError if 'a' not in dictionary
my_dict.get('a', DEFAULT_VALUE)

if 'key' in my_dict:
    val = my_dict['key']
val = my_dict.get('key', None)
if val is not None: pass
with suppress(KeyError):
    val = my_dict['key']

# iterations
for k in my_dict: pass  # or for k in my_dict.keys()
for v in my_dict.values(): pass
for k, v in my_dict.items():
    # since items() generates the items as the iteration happens, 
    #  my_dict cannot be modified in this loop.
    # For modification use tuple(my_dict.items())
    pass

# remove key from dict
del my_dict['key']  # can raise KeyError

# if you want to use the value, use .pop() and define a default 
# value to avoid KeyErrorsmy_dict.pop('key', DEFAULT_VALUE)

# set
empty_set = set()  # {} would initialize an empty dict
my_set = {1, 2, 3}
if 1 in set: pass
# there are many set methods, go check them out yourself
# some include: union, intersect, difference
# you can use + and - as well

Data Structure Usage (Efficiency)

The data structure you use is very important to writing good code.

use dictionaries if order doesn’t matter + each key has information (value) associated with ituse sets if order doesn’t matter + no values per key (e.g. keeping track of what you have ‘used’ per se)use tuples if you need ordered data but don’t need to modify the data (e.g. coordinates)use lists if you need order and mutability (most flexible)

You can’t use sets or dictionaries or sets if you need to keep track of
duplicates. That’s because sets and dictionaries hash the keys so that
it is super fast (O(1)) to check if a key is in a dictionary. This does
mean that you can’t use lists, sets, and generators as keys (but you can definitely use tuples as long as lists are not nested).

Dictionaries are also like JSON objects so you can actually use the json module to export them to a JSON file. Note that if you’re using sets as values, they are converted to lists in an exported json file.

Miscellaneous Functions

Sometimes you will see functions like

func(*args, **kwargs)

# args = a list of arguments
# kwargs = keyword arguments 
# (in the function it'll be a dictionary)
# *args: list in the function **kwargs: dict in the function
def complex_func(*args, **kwargs):
    pass

def normal_func(a, b, c, sample_param=5):
   pass

sample_args = {'sample_param': 3}
args = [0, 1, 2]

complex_func(1, 2, 3, test='true')  # how you'd call it
complex_func(*args, **sample_args)  # also works on normal functions
normal_func(*args, **sample_args)

List Comprehension and Ternary

One of the most beautiful parts of Python is list comprehensions; one liners to create lists.

# example: input is space separated integers
integers = [int(x) for x in input.split()]
# split(sep=' ', maxsplit=-1), -1 means no limit
no_negatives = [x for x in integers if x > 0]  # only if
positives = [x if x > 0 else -x for x in integers]  # if and else
back_to_str = ' '.join((str(x) for x in integers))
# items in the list to join need to be of type str
print(integers)

# this next case demonstrates the ternary operator _ if _ else _
print('list is', 'not empty' if integers else 'empty')

You can also use list comprehensions to create dictionaries and sets

set_example = {x for x in range(3)}
dict_example = {x: x for x in range(3)}
generator_example = (x for x in range(3))  # use sparingly

The third example is a generator. There are some use cases for it, so do your research before using them as they are an advanced topic not for this article.

Iterables vs. Primitives

There is one very important distinction between primitive variables and iterable variables. For example.

a = 5
b = a
a = 6
print(a == b)  # false
# vs.
a = [1, 2, 3]
b = a
c = [1, 2, 4]
a[2] = 4
print(a == b == c)  # true
print(a is b)  # true; same refrence
print(a is c)  # false

This is especially important when dealing with nested iterables with how you create nested iterables and also copy them. Try out these examples yourself.

lols = [[0] for i in range(3)] # [0] is created 3 times
lols[0][0] = 5
print(lols)  # [[5], [0], [0]]
# vs.
a = [[0]]
lols = a * 3  # same as lols = [[0] * 3]
lols[0][0] = 5
print(lols)  # [[5], [5], [5]]

Copying Iterables

To make a shallow copy, use .copy(). BUT, note that for any nested iterables, only the reference is copied, not the actual nested list. That’s why it’s called a shallow copy. To deepcopy, we can use the copy module.

new_copy = lols.copy()  # I prefer this over using [:]
reversed_list = lols[::-1]
# I rarely use this^ as reversed() and .reverse() exist
new_copy[0][0] = 6  # lols == [[6], [6], [6]]
assert lols == new_copy and not lols is new_copy

from copy import deepcopy

new_copy = deepcopy(list_of_lists)
new_copy = list_of_lists
new_copy[0][0] = 4  # [[4], [4], [4]] because 3x of the same list
assert lols != new_copy and lols is not new_copy

Memoization

Memoization is the caching of function return results in order to speed up repetitive calculations. An example would be the recursive implementation of the Fibonacci sequence.

def memo(func):  # remove print statements in a practical setting
    cache = {}

    def _helper(x): # constrained to one param (in this case)
        # you could have multiple params (x, y, ...) and then 
        # cache using tuple keys
        if x not in cache:
            print('not in cache')
            cache[x] = func(x)
        else:
            print('in cache')
        return cache[x]

    return _helper


@memo  # square = memo(square) <-- what it means
def square(x):
    return x * x

for i in range(3):
    square(i), square(i)  # second one uses cached result

An exercise is to make an addition function (a, b) that uses memoization.

Lambdas

Usually used in place of a function parameter if the calculation is short. For example, sorting.

['aa', 'Bb', 'Cc', 'dD'].sort(key=lambda string: string.upper())
[('a', 1), ('b', 0)].sort(key=lambda pair: pair[1])
sorted([('a', 1), ('b', 0)], key=lambda pair: pair[1])
max([('a', 1), ('b', 0)], key=lambda pair: pair[1])  # and min

Modules

Modules play a big part in projects you will do. Some built-in ones are os, shutil, copy, glob, and threading.

os

import os
os.mkdir()  # to make a NEW dir
os.chdir()  # choose a current working dir
os.getcwd()  # get current working dir
os.path.exists()
os.rename()
os.remove()  # for existing files only
os.rmdir()
os.getenv('key')  # gets an environmental variable

# use the shutil module for directories with sub directoriese

Environmental variables

Specify project secrets in a 

.env

# in .env
KEY=VALUE

# in *.py
with open('.env') as f:
    for line in f.read().splitlines():
        k, v = line.split('=', 1)
        os.environ[k] = v

glob

Used for getting a list of files/folders

from glob import glob

print(glob('*.py'))  # get all .py files in cwd, * is a wildcard
# exercise: find out how to get all .py files in cwd + its subdirs

Threading

I have already written some threading example for you to look at here. Just follow the instructions in the gist.

Third Party Modules

You need to use `pip install module_name` to install modules.

Some modules that are common are requests, beautifulsoup4, PIL, and flask. If you’re working on a big project, you’ll probably end up using 3rd party modules.

Advanced Topics (future Python learning)

Classes

I did not cover classes because that is more about OOP than Python programming and the use cases for classes are very small. One thing you should know when you are learning classes is __slots__ property, so do search that up on your own.

Generators

Again this is an advanced topic and learning about it now will only lead to confusion, its best to learn this on your own or in a practical setting.

Decorators

I covered only the basics of decorators. There are decorators used by lots of other 3rd party libraries and different use cases (e.g. timing functions) so I suggest you do your own research on them as well.

git and git workflow

This is very important when your collaborating with others or are working for a company. Git is a versioning tool used so that mistakes don’t hurt you, and for letting you work on multiple features at the same time.

other builtin modules

Such as itertools, threading, multiprocessing, and more.

Project Ideas

You can find a list of project ideas below:

PRACTICAL PROJECT IDEAS

Your own website/portfolio (no bootstrap, use raw HTML/CSS/JS + flask backend). Use Heroku (free) to host it [Hard]

A program that applies a blur to an image you select [Easy]
    (use tkinter/SG so that you don't have to hard code the filename every time someone uses the program)

A program that sets your desktop wallpaper and cycles through a folder of images, [Okay]
    change the wallpaper every x (you decide) seconds.

Web Scraping & Data Parsing [Hard]
    Go to https://www.cia.gov/library/publications/the-world-factbook/rankorder/2004rank.html and use inspect elements,
    Your job is to parse this web page and filter the data so that only the data with the latest "date of information"
    remains. Output should be order-preserved and in a CSV file (filename is arbitrary)
    To make it easier (not recommended), you can assume the latest date of info is 2017 EST.
    To make it even easier (not recommended) just preserve the order and output all of the data in a CSV
    hint: use requests, bs4, and the csv module
    Another challenge is to download the data as a txt file and convert that text file to a CSV file using Python
    hint: you may find ' '.join(iterable) useful

Using an API [Okay - Hard]
    Spotify Reference
    You'll have to get a Spotify API key
    See https://developer.spotify.com/documentation/web-api/reference/search/search/ and
    make a program that lets users search for tracks using the Spotify API (you return a list of tracks)
    See https://elijahlopez.herokuapp.com/search-album-art/ for a demo but with album art
    You just need to spit out a list of tracks
    To make this harder, you will sort the tracks by popularity (you'll have to search the documentation yourself)
    To make this even harder, you can add a sort by artists feature and MORE
    # requests, base64 module,

Try to send an email to yourself (your email -> your email) using Python [Easy - Hard]
    To make this harder, try to also send an attachment
    module: at least smptlib

You can try to make your own key logger [Okay - Hard]
    At first just append the contents to a .log or .txt file
        Use the 'a' open type to append to bottom.
        HARDER: YOU WILL INCLUDE COPIED AND PASTED ITEMS WITH custom syntax
        # (e.g. USER_COPIED("")), (e.g. USER_PASTED(""))

Make your own soundboard with your own sounds [Hard because tedious]
    You PySimpleGUI for the GUI (I recommend but you don't have to)

    - Let your soundboard include support for different "packs"
    - Specify a design/package guide for other users to create use their sounds on your soundboard
    default: your default pack
    PACKS_DIR/pack_name/sound_name/sound .mp3/ogg
    # pygame, playsound


Make your own game using pygame (OOP)


Turn one of these projects into an executable [Okay but Tedious]
    - use sys.exit() instead of quit()
    pyinstaller, or cx_freeze

Make your own discord bot [variable difficulty] (async)

Email me or comment for bigger programming (not specifically Python) project ideas

Comment if you have any questions.

Thanks for reading and good luck to your learning journey.