Advanced Python Concepts Lecture by Patrick

Target Audience

• Intermediate Python programmers.
• People who have basic Python knowledge.

Course Outline

• 21 sections in total.
• Detailed exploration of Python's advanced topics.
• Sharing helpful tips throughout the course.

Topics Covered Today

Lists

• Definition: Ordered, mutable collections allowing duplicate elements.
• Creation: Use square brackets [] or the list() function.
• Examples: ['banana', 'cherry', 'apple'], list() -> []
• Data Types: Lists can contain mixed data types.
• Duplicate Elements: Allowed in lists.
• Access Elements: Use indices, starting from 0.
• Negative Indices: Accesses items from the end (-1 for last item).
• Iterate Over Lists: Use `for item in list:
• Check Membership: Use if item in list: ...
• Useful Methods:
  • len(list): Get number of elements.
  • list.append(item): Add an element at the end.
  • list.insert(index, item): Insert element at a specific position.
  • list.pop(): Remove and return the last element.
  • list.remove(item): Remove first occurrence of an element.
  • list.clear(): Remove all elements.
  • list.reverse(): Reverse the order of elements.
  • list.sort(): Sort the list in ascending order.
  • sorted(list): Return a new sorted list without changing the original.
• Tricks:
  • Create list with repeated elements: [0] * 5 -> [0, 0, 0, 0, 0]
  • Concatenate lists: list1 + list2
  • Slicing lists: list[start:stop:step]
  • Copy lists: list.copy(), list(list), slicing [:]
  • Advanced: List comprehensions for creating new lists in a concise way.*`

Tuples

• Definition: Ordered, immutable collections.
• Creation: Use parentheses () or the tuple() function.
• Single Element Tuple: Needs a trailing comma, e.g., (item,)
• Access Elements: Use indices, starting from 0.
• Negative Indices: Accesses items from the end (-1 for last item).
• Iterate Over Tuples: Use for item in tuple: ...
• Check Membership: Use if item in tuple: ...
• Methods:
  • len(tuple): Get number of elements.
  • tuple.count(item): Count occurrences of an element.
  • tuple.index(item): Find first occurrence of an element.
• Convert Between Tuples and Lists: list(tuple), tuple(list)
• Slicing Tuples: tuple[start:stop:step]
• Tuple Unpacking: Assign values from a tuple to variables.
• Memory and Efficiency: Tuples are more memory-efficient than lists.

Dictionaries

• Definition: Unordered, mutable collections of key-value pairs.
• Creation: Use braces {} or the dict() function.
• Access Elements: Use keys, not indices.
• Add/Change Items: Use dict[key] = value
• Delete Items:
  • del dict[key]
  • dict.pop(key)
  • dict.popitem(): Pops last item (Python 3.7+).
• Check Membership: Use if key in dict: ...
• Loop Through Keys, Values, Items:
  • for key in dict: ... or dict.keys()
  • for value in dict.values(): ...
  • for key, value in dict.items(): ...
• Copy Dictionaries: Use dict.copy() or dict(dict)
• Merge Dictionaries: Use dict1.update(dict2)
• Possible Key Types: Immutable types like strings, numbers, tuples.
• Efficiency: Compare size and iterate time of dicts vs lists.

Sets

• Definition: Unordered, mutable collections of unique elements.
• Creation: Use braces {} or the set() function.
• Duplicate Elements: Are automatically removed.
• Basic Operations: add(item), remove(item), discard(item), clear(), pop()
• Iterate Over Sets: Use for item in set: ...
• Check Membership: Use if item in set: ...
• Set Operations:
  • Union: set1.union(set2) or |
  • Intersection: set1.intersection(set2) or &
  • Difference: set1.difference(set2) or -
  • Symmetric Difference: set1.symmetric_difference(set2) or ^
• Subset/Superset: issubset(), issuperset()
• Copy Sets: Use set.copy() or set(set)
• Special Sets: frozenset (immutable set)

Strings

• Definition: Ordered, immutable collection of characters.
• Creation: Use single '' or double "" quotes for strings, or triple quotes for multi-line strings.
• Access Characters: Use indices.
• Concatenation: Use + operator.
• Iterate Over Strings: Use for char in string: ...
• Check Membership: Use if char in string: ...
• Useful Methods: .upper(), .lower(), .startswith(), .endswith(), .find(), .count(), .replace()
• Split/Join Strings: string.split(separator), 'separator'.join(list)
• String Formatting:
  • Old Style: '%s %d' % (str, int)
  • str.format(): '{0} {1}'.format(str, int)
  • F-strings (Python 3.6+): f'{str} {int}'

Collections Module

• Special Data Types:
  • Counter: Count hashable objects.
  • NamedTuple: Tuple with named fields.
  • OrderedDict: Dictionary that remembers the order of items.
  • DefaultDict: Dictionary with default values for missing keys.
  • Deque: Double-ended queue.

Itertools Module

• Functions:
  • Product: Cartesian product of input iterables.
  • Permutations: All possible orderings of input.
  • Combinations: All possible combinations of input.
  • Accumulate: Return accumulated sums or other binary functions.
  • Groupby: Group data.
  • Infinite Iterators: Count, cycle, repeat.

Lambda Functions

• Definition: Small anonymous functions defined with lambda keyword.
• Syntax: lambda arguments: expression
• Use Case: Functions as arguments to higher-order functions like sorted(), map(), filter(), reduce().

Exception Handling

• Types: Syntax errors vs exceptions.
• Common Exceptions: ImportError, NameError, FileNotFoundError, ValueError, IndexError, KeyError
• Raise Exceptions: Use raise keyword.
• Assert Statements: assert condition, message
• Try-Except Block:
  • Basic: try: ... except: ...
  • Specific Exception: try: ... except ExceptionType: ...
  • Else and Finally: else: ..., finally: ...
• Custom Exceptions: Create by subclassing built-in Exception.

Logging Module

• Setup Basic Logging: import logging, logging.basicConfig(level=logging.DEBUG)
• Log Levels: DEBUG, INFO, WARNING, ERROR, CRITICAL
• Handlers: StreamHandler, FileHandler
• Config Files: Use .conf files for configuration.
• Capture Tracebacks: exc_info=True
• RotatingFileHandler: Handle large log files with rotation.

JSON Module

• Encode/Decode JSON:
  • Use json.dumps(), json.dump() for encoding.
  • Use json.loads(), json.load() for decoding.
• Handle Custom Objects:
  • Custom encoding with default function.
  • Custom decoding with object_hook.

Random Module

• Random Numbers:
  • Basics: random.random(), random.uniform(start, stop), random.randint(start, stop).
  • Advanced: random.shuffle(list), random.sample(list, k), random.choices(list, k), random.choice(list).
• Secrets Module: Cryptographically strong random numbers.
• NumPy Random: Generate random arrays with NumPy.

Decorators

• Concept: Functions modifying another function’s behavior.
• Syntax: @decorator above a function definition.
• Function Decorators: Define with inner function wrapper.
• Class Decorators: Define with __call__ method.
• Use Cases: Logging, enforcing types, memoization.

Generators

• Definition: Functions returning an iterator with yield keyword.
• Usage: Generating values on-the-fly, memory efficiency.
• Generator Expressions: Compact version of generators using parentheses.

Threading and Multiprocessing

• Threading:
  • Creation: from threading import Thread
  • Start and Join: thread.start(), thread.join()
  • Locks: Ensure safe access to shared resources.
• Multiprocessing:
  • Creation: from multiprocessing import Process
  • Start and Join: process.start(), process.join()
  • Shared Memory: Use Value, Array for shared state.
  • Queues: For safe inter-process communication.
  • Pool: Manage multiple processes efficiently.

Function Arguments

• Positional and Keyword Arguments: Pass arguments by position or name.
• Default Arguments: Specify default values for parameters.
• Variable-Length Arguments: Use *args for positional, **kwargs for keyword arguments.
• Unpacking Arguments: Use * and ** for unpacking iterables into function calls.

Asterisk Operator

• Multiplication and Power: Use * for multiplication, ** for power.
• Create Lists/Strings: Repeat elements using *.
• Function Arguments: Use *args and **kwargs.
• Unpack Containers: Use * and ** to unpack lists, tuples, dicts.
• Merging Containers: Merge lists, tuples, sets with *, merge dicts with **.*

Copying

• Assignment Operator: Does not make a copy, just a new reference.
• Shallow vs Deep Copy: Use copy.copy() for shallow, copy.deepcopy() for deep copy.
• Custom Objects: Implement copying by defining __copy__ and __deepcopy__ methods.

Context Managers

• Concept: Ensure resources are properly managed with with statement.
• Typical Examples: with open(file), file operations, with lock, managing locks.
• Custom Context Managers:
  • Class-Based: Define __enter__ and __exit__ methods.
  • Function-Based: Use contextlib.contextmanager decorator with a generator function.