Intermediate Python Refresher: Tutorial, Project Ideas, and Tips

This article is to teach Python beginners and developers some key concepts used in Python that aren't taught from the get-go. Last updated: May 17th, 2020 If you can create a quadratics root solver, you'll be able to understand this article. These are some concepts I didn't learn in one day but rather a couple years as I am self-taught. I tried to make this into a video, but the video ended up being 1:45 hours long; I myself would need a lot of motivation to watch a “tutorial” that long, and I prefer articles to get my information since I can pick out relevant details rather than trying to skip sections of a video. How It All Started I learned Python basics through and then proceeded to improve/test my problem solving skills. I did this by doing questions which you can find (among other contest problems) at . Other sites to improve your algorithmic problem solving skills include and (although this is mainly for interview preparations). CS Circles, CCC DMOJ HackerRank LeetCode While I was doing this, I was coding with the default IDLE editor! I did this for 3 months and then I found out about which has a slight learning curve but is loads better in terms of features and at improving productivity. Nowadays, I use both PyCharm and . PyCharm Visual Studio Code I personally have an entire folder dedicated to snippets of code I could use in the future, so I suggest you also do that and maybe even add some of the code snippets in this article in there so that you can read your own examples instead of Googling them or coming back to this article. General Tips These are some tips that are not bound to programming but just life and productivity in general. Know both the program specific ones (browser, explorer, IDE of choice, etc.) and also OS specific ones (e.g. Win + R for run). Know your keyboard shortcuts Know the command line Instead of doing a calculation by hand or opening an IDE to create and run a script, you can actually execute Python code from the command line. Aside from the common batch functions (e.g. ls, cd), knowing how to use Python from the command line will save you a lot of time Command Line Python Example Know how to Google Google (or your search engine of choice), is my best friend and should also be yours. It has saved me a lot of time and so it could also save you a lot of time. It can’t do that if you don’t use it or don’t know how to use it. When you Google something, your query needs to be general enough that you can find answers, but also specific enough so that those answers are relevant. Problem Breakdown Strategy This goes hand in hand with Googling. Suppose you have a problem/project. You need to break it down into smaller parts. You then need to analyze each of these parts and see if they are small enough for you to complete each of them. If not, either your missing some knowledge that you should Google or the part is too big and needs to be broken down again. You keep doing this recursive procedure until your project has been split into solvable parts so that you can complete them and then weave together a project. When I search and find answers through Google, I don’t expect them to be 100% what I need. I usually need to them into what I want and that’s what you should also expect: the bare minimum solution that takes you at least one step forward. remix With these tips stated, you can do a couple of different things next. You can skim the rest of the document and make notes on the snippets of code I feature (what I would do personally), read only the headings, skip to the project ideas section, or stop reading altogether as my tips are so useful. Refresher In CS Circles, they bring abou`t the print function and some of its optional parameters but it’s easy to forget about them so here it is again. print( , , , sep= , end= ) / / --------- >>> # The default parameters for print are sep=' ', and end='\n' >>> '4' '21' 2020 '/' '\n---------\n' 4 21 2020 Let’s Start Input function and String Formatting The input function has an optional parameter so that it can also act as a prompt and if you are using Python 3.6+, you can make use of f-strings. name = input( ) print( ) 'Enter your name: ' f'Hello !' {name} # modern way of string formatting # if input='reader', output: Hello reader! I want to make clear to you that a for loop, is not a while loop as it is in other languages. In Python, a for loop is an iteration over an iterable object. For Loops The range function has three parameters, two of them being optional. Do not write use the range function with an explicit start value of 0 because 0 is the default start value, (unless of course you are modifying the default step value of 1). In this example, I will show you exactly what I mean by “not a while loop” and how a for loop (specifically range) does not add to the temporary value. i range( ): print(i) i += # range(start=0, stop, step=1) # range(5) == range(0, 5) == range(0, 5, 1) for in 5 2 # Guess the output. HINT: i += 2does nothing If you run this code, you’ll notice that the output is increasing by 1 each time even if we are adding 2 to i at every loop. This is because is set to the next value in range and isn’t actually being increased by one each time. This means that we can actually iterate over all sorts of iterable objects, like lists, without having to use range and indexing. i some_letters = [ , , , , ] letter some_letters: 'a' 'b' 'c' 'd' 'e' for in # do something pass Here I introduced the keyword pass, this is to avoid errors in otherwise empty blocks. If you do want to keep track of the index as well as the item, you still don’t have to use range, you can use the built-in function enumerate. i, letter enumerate(some_letters, start= ): print( ) for in 0 f'item at index is ' {i} {letter} You can think of enumerate as turning an iterable into an iterable of pairs (index, item of iterable at index). You can also use the next function to retrieve the next value in an iterable (if there is no next item, an error will be raised). File I/O I’m including this because if you search “how to read files in python” on Google, you are given which teaches it the old way and not the modern approach. this f = open( ) f.close() open( ) f: print(f.read()) f.read() f.seek( ) f.read() open( , ) f: f.write( ) f.writelines([ , ]) # make sure test.txt exists with text in it # OLD 'test.txt' # note default is mode='r' # do something with f here with 'test.txt' as # NEW; no close() needed # f.read() moves the "cursor" to the end of the file assert not 0 assert # f.read() returns a string now (unless test.txt is empty) with 'test.txt' 'w' as # f.read() ERROR do not do this 'this is a test\n' # note there is no end parameter 'line1\n' 'line2\n' # note no auto newline# other modes: a for append, rb for reading-bytes, wb for writing bytes, and r+/w+ for both reading and writing at the same time If you are curious why the r+/w+ is not the default, think about how a file cannot be opened to be written to if it is being “written” to by another program. If you just need to read a file, it being open in another program means that you won’t be interfering with the other program. Error Handling : RuntimeWarning( ) RuntimeWarning e: : Exception( ) Exception: contextlib suppress suppress(exception): exception( ) print( ) ignore_error(RuntimeError) print( ) # handling the error try raise 'Something could go wrong' except as # as e is optional # handle the exception here # ignoring the error # old try raise 'BOO' except pass # new from import : def ignore_error (exception: Exception) """ Use three quotes for docstrings or long strings """ # : for type hinting (in a dynamic typed language!) and # yes you can pass exceptions and functions as parameters with raise 'BOO' 'not printed' 'this gets printed' By this point if you are following along in PyCharm, you would have seen some squiggly lines, especially under “Exception” in the above code. These squiggly lines help you to avoid syntax errors, follow style guidelines, and bring attention to code that could be doing something you didn’t want it to be doing. More Data Types So what are these iterables I keep mentioning? Yes a list is an iterable and so are tuples (which you should already know of). There are also dictionaries, sets and generators (not discussed here). Dictionaries are like hash tables in other languages, because they “hash” the key to store information. empty_dict = {} # or dict() my_dict = { : } # How to get value dict my_dict[ ] # raises KeyError not dictionary my_dict.get( , DEFAULT_VALUE) my_dict: val = my_dict[ ] val = my_dict.get( , None) val is not None: pass suppress(KeyError): val = my_dict[ ] # iterations k my_dict: pass # or k my_dict.keys() v my_dict.values(): pass k, v my_dict.items(): # since items() generates the items the iteration happens, # my_dict cannot be modified loop. # For modification use tuple(my_dict.items()) pass # remove key dict del my_dict[ ] # can raise KeyError # you want to use the value, use .pop() and define a # value to avoid KeyErrorsmy_dict.pop( , DEFAULT_VALUE) # set empty_set = set() # {} would initialize an empty dict my_set = { , , } set: pass # there are many set methods, go check them out yourself # some include: union, intersect, difference # you can use + and - well 'key' 'value' from 'a' if 'a' in 'a' if 'key' in 'key' 'key' if with 'key' for in for in for in for in as in this from 'key' if default 'key' 1 2 3 if 1 in as The data structure you use is very important to writing good code. Data Structure Usage (Efficiency) use dictionaries if order doesn’t matter + each key has information (value) associated with ituse sets if order doesn’t matter + no values per key (e.g. keeping track of what you have ‘used’ per se)use tuples if you need ordered data but don’t need to modify the data (e.g. coordinates)use lists if you need order and mutability (most flexible) You can’t use sets or dictionaries or sets if you need to keep track of duplicates. That’s because sets and dictionaries hash the keys so that it is super fast (O(1)) to check if a key is in a dictionary. This does mean that you can’t use lists, sets, and generators as keys (but you can definitely use tuples as long as lists are not nested). Dictionaries are also like JSON objects so you can actually use the json module to export them to a JSON file. Note that if you’re using sets as values, they are converted to lists in an exported json file. Miscellaneous Functions Sometimes you will see functions like func(*args, **kwargs) sample_args = { : } args = [ , , ] complex_func( , , , test= ) complex_func(*args, **sample_args) normal_func(*args, **sample_args) # args = a list of arguments # kwargs = keyword arguments # (in the function it'll be a dictionary) # *args: list in the function **kwargs: dict in the function : def complex_func (*args, **kwargs) pass : def normal_func (a, b, c, sample_param= ) 5 pass 'sample_param' 3 0 1 2 1 2 3 'true' # how you'd call it # also works on normal functions List Comprehension and Ternary One of the most beautiful parts of Python is list comprehensions; one liners to create lists. integers = [int(x) x input.split()] no_negatives = [x x integers x > ] positives = [x x > -x x integers] back_to_str = .join((str(x) x integers)) print(integers) print( , integers ) # example: input is space separated integers for in # split(sep=' ', maxsplit=-1), -1 means no limit for in if 0 # only if if 0 else for in # if and else ' ' for in # items in the list to join need to be of type str # this next case demonstrates the ternary operator _ if _ else _ 'list is' 'not empty' if else 'empty' You can also use list comprehensions to create dictionaries and sets set_example = {x x range( )} dict_example = { : x x range( )} generator_example = (x x range( )) # use sparingly for in 3 x for in 3 for in 3 The third example is a generator. There are some use cases for it, so do your research before using them as they are an advanced topic not for this article. Iterables vs. Primitives There is one very important distinction between primitive variables and iterable variables. For example. a = b = a a = print(a == b) a = [ , , ] b = a c = [ , , ] a[ ] = print(a == b == c) print(a b) print(a c) 5 6 # false # vs. 1 2 3 1 2 4 2 4 # true is # true; same refrence is # false This is especially important when dealing with nested iterables with how you create nested iterables and also them. Try out these examples yourself. copy lols = [[ ] i range( )] lols[ ][ ] = print(lols) a = [[ ]] lols = a * lols[ ][ ] = print(lols) 0 for in 3 # [0] is created 3 times 0 0 5 # [[5], [0], [0]] # vs. 0 3 # same as lols = [[0] * 3] 0 0 5 # [[5], [5], [5]] Copying Iterables To make a shallow copy, use .copy(). BUT, note that for any nested iterables, only the reference is copied, not the actual nested list. That’s why it’s called a shallow copy. To , we can use the copy module. deepcopy new_copy = lols.copy() reversed_list = lols[:: ] new_copy[ ][ ] = lols == new_copy lols new_copy copy deepcopy new_copy = deepcopy(list_of_lists) new_copy = list_of_lists new_copy[ ][ ] = lols != new_copy lols new_copy # I prefer this over using [:] -1 # I rarely use this^ as reversed() and .reverse() exist 0 0 6 # lols == [[6], [6], [6]] assert and not is from import 0 0 4 # [[4], [4], [4]] because 3x of the same list assert and is not Memoization Memoization is the caching of function return results in order to speed up repetitive calculations. An example would be the recursive implementation of the Fibonacci sequence. cache = {} x cache: print( ) cache[x] = func(x) : print( ) cache[x] _helper x * x i range( ): square(i), square(i) : def memo (func) # remove print statements in a practical setting : def _helper (x) # constrained to one param (in this case) # you could have multiple params (x, y, ...) and then # cache using tuple keys if not in 'not in cache' else 'in cache' return return @memo # square = memo(square) <-- what it means : def square (x) return for in 3 # second one uses cached result An exercise is to make an addition function (a, b) that uses memoization. Lambdas Usually used in place of a function parameter if the calculation is short. For example, sorting. [ , , , ].sort(key= string: string.upper()) [( , ), ( , )].sort(key= pair: pair[ ]) sorted([( , ), ( , )], key= pair: pair[ ]) max([( , ), ( , )], key= pair: pair[ ]) 'aa' 'Bb' 'Cc' 'dD' lambda 'a' 1 'b' 0 lambda 1 'a' 1 'b' 0 lambda 1 'a' 1 'b' 0 lambda 1 # and min Modules Modules play a big part in projects you will do. Some built-in ones are os, shutil, copy, glob, and threading. os os os.mkdir() # to make a NEW dir os.chdir() # choose a current working dir os.getcwd() # get current working dir os.path.exists() os.rename() os.remove() # existing files only os.rmdir() os.getenv( ) # gets an environmental variable # use the shutil directories sub directoriese import for 'key' module for with Environmental variables Specify project secrets in a file .env KEY=VALUE open( ) f: line f.read().splitlines(): k, v = line.split( , ) os.environ[k] = v # in .env # in *.py with '.env' as for in '=' 1 glob Used for getting a list of files/folders glob glob print(glob( )) from import '*.py' # get all .py files in cwd, * is a wildcard # exercise: find out how to get all .py files in cwd + its subdirs Threading I have already written some threading example for you to look at . Just follow the instructions in the gist. here Third Party Modules You need to use `pip install module_name` to install modules. Some modules that are common are requests, beautifulsoup4, PIL, and flask. If you’re working on a big project, you’ll probably end up using 3rd party modules. Advanced Topics (future Python learning) Classes I did not cover classes because that is more about OOP than Python programming and the use cases for classes are very small. One thing you should know when you are learning classes is property, so do search that up on your own. __slots__ Generators Again this is an advanced topic and learning about it now will only lead to confusion, its best to learn this on your own or in a practical setting. Decorators I covered only the basics of decorators. There are decorators used by lots of other 3rd party libraries and different use cases (e.g. timing functions) so I suggest you do your own research on them as well. git and git workflow This is very important when your collaborating with others or are working for a company. Git is a versioning tool used so that mistakes don’t hurt you, and for letting you work on multiple features at the same time. other builtin modules Such as itertools, threading, multiprocessing, and more. Project Ideas You can find a list of project ideas below: PRACTICAL PROJECT IDEAS Your own website/portfolio (no bootstrap, use raw HTML/CSS/JS + flask backend). Use Heroku (free) to host it [Hard] A program that applies a blur to an image you select [Easy] (use tkinter/SG so that you don ll have to get a Spotify API key See https: make a program that lets users search tracks using the Spotify API (you a list tracks) See https: You just need to spit out a list tracks To make harder, you will sort the tracks by popularity (you a t have to) - Let your soundboard include support different - Specify a design/package guide other users to create use their sounds on your soundboard : your pack PACKS_DIR/pack_name/sound_name/sound .mp3/ogg # pygame, playsound Make your own game using pygame (OOP) Turn one these projects into an executable [Okay but Tedious] - use sys.exit() instead quit() pyinstaller, or cx_freeze Make your own discord bot [variable difficulty] ( ) Email me or comment bigger programming (not specifically Python) project ideas 't have to hard code the filename every time someone uses the program) A program that sets your desktop wallpaper and cycles through a folder of images, [Okay] change the wallpaper every x (you decide) seconds. Web Scraping & Data Parsing [Hard] Go to https://www.cia.gov/library/publications/the-world-factbook/rankorder/2004rank.html and use inspect elements, Your job is to parse this web page and filter the data so that only the data with the latest "date of information" remains. Output should be order-preserved and in a CSV file (filename is arbitrary) To make it easier (not recommended), you can assume the latest date of info is 2017 EST. To make it even easier (not recommended) just preserve the order and output all of the data in a CSV hint: use requests, bs4, and the csv module Another challenge is to download the data as a txt file and convert that text file to a CSV file using Python hint: you may find ' '.join(iterable) useful Using an API [Okay - Hard] Spotify Reference You' //developer.spotify.com/documentation/web-api/reference/search/search/ and for return of //elijahlopez.herokuapp.com/search-album-art/ for a demo but with album art of this 'll have to search the documentation yourself) To make this even harder, you can add a sort by artists feature and MORE # requests, base64 module, Try to send an email to yourself (your email -> your email) using Python [Easy - Hard] To make this harder, try to also send an attachment module: at least smptlib You can try to make your own key logger [Okay - Hard] At first just append the contents to a .log or .txt file Use the ' ' open type to append to bottom. HARDER: YOU WILL INCLUDE COPIED AND PASTED ITEMS WITH custom syntax # (e.g. USER_COPIED("")), (e.g. USER_PASTED("")) Make your own soundboard with your own sounds [Hard because tedious] You PySimpleGUI for the GUI (I recommend but you don' for "packs" for default default of of async for Comment if you have any questions. Thanks for reading and good luck to your learning journey.