Dabeaz

For the past few years, I've been making my PyCon tutorials and presentations available online. For example, Generator Tricks for Systems Programmers from PyCon'2008, A Curious Course on Coroutines and Concurrency from PyCon'2009, and Mastering Python 3 I/O from PyCon'2010. Although there have been many downloads, I've occasionally received requests to post material in a format more suitable for sharing online.

Thus, I'm pleased to announce that I've set up a Slideshare channel that has the slides from almost all my past presentations and tutorials from PyCon, the International Python Conference, USENIX, and a few other conferences, going all the way back to 1996. All told, there are more than 1700 slides on Python programming, Swig, PLY, and other topics.

I hope someone finds this material useful so enjoy! I'm still going through my presentation archive and will probably add even more to Slideshare as I find time.

-- Dave

posted by Dave Beazley  # 6:25 PM

This fall, I am offering three intense Python courses in Chicago:

• Practical Python Programming, October 25-28, 2010.
• Advanced Python Mastery, November 8-11, 2010. (Only two slots left!)
• Practical Python Programing plus Django, November 15-19, 2010.

Here are some reasons you might want to attend:

1. Courses are held in a certifiably "evil" Python programming lair. Aside from some occasional C and assembly hacking, this is where I do all of my Python programming. Want to take a class to go get "certified" in some kind of "enterprise" software or Microsoft Office? Bah. Better look elsewhere. Python is my only focus here.
   

2. Be like a rocket scientist. These are the same Python classes I regularly teach on-site to scientists, engineers, and yes, rocket scientists--who think Python is pretty useful by the way. However, do you have to be an expert to attend? Nope. These courses are for anyone who wants to learn more--including programmers new to Python.
3. You'll learn some new tricks for making your code better.
Even if you've been programming in Python for awhile, you will learn some new techniques. This is because I spend most of my free time exploring different ways to effectively use Python's various features--often in preparation for future writing projects, PyCon tutorials, or for use in my own coding projects. And after you've mastered everything there is to know about Python, you can move on to mastering the Curta.
4. You'll be well fed. These courses aren't held in some sterile hotel or corporate training center. The lair is surrounded by great restaurants, cafes, and bakeries. For instance, you probably don't want to know how many calories are in this picture (from the bakery located immediately below the lair):
   

5. All Python, All Day. You're going to spend several days doing nothing but hacking and talking about Python with people who like Python as much as you do. What's not to like about that?

That is all for now. Hopefully you'll join me for a future course!

--Dave

posted by Dave Beazley  # 9:05 AM

Welcome to part 3 of my "Superboard II" trilogy. For the first two parts, see these posts:

• Using Python to Encode Cassette Recordings for my Superboard II
• Decoding Superboard II Cassette Audio Using Python 3, Two Generators, and a Deque

Dave's Superboard II

First, a brief digression.

Why Bother?

Aside from the obvious nostalgia (the Superboard II being my first computer), why bother messing around with something like this? After all, we're talking about a long-since-dead 1970s technology. Any sort of practical application certainly seems far-fetched.

The simple answer is that doing this sort of thing is fun--fun for the same reasons I got into programming in the first place. When my family first got the Superboard, it was this magical device--a device where you could command it to do anything you wanted. You could write programs to make it play games. Or, more importantly, you could command it to do your math homework. Not only that, everything about the machine was open. It came with electrical schematics and memory maps. You could directly input hex 6502 opcodes. There were no rules at all. Although writing a game or doing your homework might be an end goal, the real fun was the process of figuring out how to do those things (to be honest, I think I learned much more about math by writing programs to do my math homework than I ever did by actually doing the homework, but that's a different story).

Flash forward about 30 years and I'm now doing most of my coding in Python. However, Python (and most other dynamic languages) embody everything that was great about my old Superboard II. For instance, the instant gratification of using the interactive interpreter to try things out. Or, the complete freedom to do almost anything you want in a program (first-class functions, duck-typing, metaprogramming, etc.). Or, the ability to dig deep into the bowels of your system (ctypes, Swig, etc.). Frankly, it's all great fun. It's what programming should be about. Clearly the designers of more "serious" languages (especially those designed for the "enterprise") never had anything like a Superboard.

Anyways, getting back to my motivations, I don't really have any urgent need to access my Superboard from my Mac. I'm mostly just interested in the problem of how I would do it. The fun is all in the process of figuring it out.

Back to the Superboard Cassette Ports

Getting back to topic, you will recall that in my prior posts, I was interested in the problem of encoding and decoding the audio stream transmitted from the cassette input and output ports on my Superboard II. In part, this was due to the fact that those are the only available I/O ports--forget about USB, Firewire, Ethernet, RS-232, or a parallel port. Nope, cassette audio is all there is.

From the two parts, I wrote some Python scripts that encode and decode the cassette audio data to and from WAV files. Although that is somewhat interesting, working with WAV files was never my real goal. Instead, what I really wanted to do was to set up a real-time bidirectional data communication channel between my Mac and the Superboard II. Simply stated, I wanted to create the equivalent of a network connection using the cassette ports. Would it even be possible? Who knows?

So far as I know, the cassette ports on the Superboard were never intended for this purpose. Although there are commands to save a program and to load a program, driving both the cassette input and output simultaneously isn't something you would do. It didn't even make any sense. There certainly weren't any Superboard commands to do that.

Building a Soft-Modem Using PyAudio

To perform real-time communications, the Superboard needs to be connected to both the audio line-out and line-in ports of my Mac. Using those connections, I would then need to write a program that operates as a soft-modem. This program would simultaneously read and transmit audio data by encoding or decoding it as appropriate (see my past posts).

I've never written a program for manipulating audio on my Mac, but after some searching, I found the PyAudio extension that seemed to provide the exact set of features I needed.

To create a soft-modem, I defined reader and writer threads as follows:

# Note : This is Python 2 due to the PyAudio dependency.
import pyaudio
import kcs_decode      # See prior posts
import kcs_encode      # See prior posts
from Queue import Queue

FORMAT    = pyaudio.paInt8
CHANNELS  = 1
RATE      = 9600
CHUNKSIZE = 1024

# Buffered data received and waiting to transmit
audio_write_buffer = Queue()
audio_read_buffer = Queue()

# Generate a sequence representing sign change bits on the real-time
# audio stream (needed as input for decoding)
def generate_sign_change_bits(stream):
    previous = 0
    while True:
        frames = stream.read(CHUNKSIZE)
        if not frames:
            break
        msbytes = bytearray(frames)
        # Emit a stream of sign-change bits
        for byte in msbytes:
            signbit = byte & 0x80
            yield 1 if (signbit ^ previous) else 0
            previous = signbit

# Thread that reads and decodes KCS audio input
def audio_reader():
    print("Reader starting")
    p = pyaudio.PyAudio()
    stream = p.open(format = FORMAT,
                    channels = CHANNELS,
                    rate = RATE,
                    input=True,
                    frames_per_buffer=CHUNKSIZE)

    bits = generate_sign_change_bits(stream)
    byte_stream = kcs_decode.generate_bytes(bits, RATE)
    for b in byte_stream:
        audio_read_buffer.put(chr(b))

# Thread that writes KCS audio data
def audio_writer():
    print("Writer starting")
    p = pyaudio.PyAudio()
    stream = p.open(format = FORMAT,
                    channels = CHANNELS,
                    rate = RATE,
                    output=True)
    while True:
        if not audio_write_buffer.empty():
            msg = kcs_encode.kcs_encode_byte(ord(audio_write_buffer.get()))
            stream.write(buffer(msg))
        else:
            stream.write(buffer(kcs_encode.one_pulse))

if __name__ == '__main__':
    import threading

    # Launch the reader/writer threads
    reader_thr = threading.Thread(target=audio_reader)
    reader_thr.daemon = True
    reader_thr.name = "Reader"
    reader_thr.start()

    writer_thr = threading.Thread(target=audio_writer)
    writer_thr.daemon = True
    writer_thr.name = "Writer"
    writer_thr.start()    

The operation of this code is relatively straightforward. There is a reader thread that constantly samples audio on the line-in port and decodes it into bytes which are stored in a queue for later consumption. There is a writer thread that encodes and transmits outgoing data (if any). If there is no data, the writer transmits a constant carrier tone on the line out (a 2400 Hz wave).

These threads operate entirely in the background. To read data from the Superboard, you simply check the contents of the audio read buffer. To send data to the Superboard, you simply append outgoing data to the audio write buffer.

Creating a Network Server

To tie all of this together, you can now write a network server that connects the real-time audio streams to a network socket. This can be done by defining a third thread like this:

import socket
import time

def server(addr):
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR,1)
    s.bind(addr)
    s.listen(1)
    print("Server running on", addr)
    # Wait for the client to connect
    while True:
        c,a = s.accept()
        print("Got connection",a)
        c.setblocking(False)
        try:
            # Enter a loop where we try to transmit data back and forth between the client and the audio stream
            while True:
                # Check for incoming data
                try:
                    indata = c.recv(8192)
                    if not indata:
                        raise EOFError()
                    indata = indata.replace(b'\r',b'\r' + b'\x00'*20)
                    for b in indata:
                        audio_write_buffer.put(b)
                except socket.error:
                    pass
                # Check if there is any outgoing data to transmit (try to send it all)
                if not audio_read_buffer.empty():
                    while not audio_read_buffer.empty():
                        b = audio_read_buffer.get()
                        c.send(b)
                else:
                    # Sleep briefly if nothing is going on.  This is fine, the max
                    # data transfer rate of the Superboard is 300 baud
                    time.sleep(0.01)
        except EOFError:
            print("Connection closed")
            c.close()

if __name__ == '__main__':
    import threading

    # Launch the reader/writer threads
    ... see above code ..

    # Launch the network server
    server_thr = threading.Thread(target=server,args=(("",15000),))
    server_thr.daemon = True
    server_thr.name = "Server"
    server_thr.start()

    # Have the main thread do something (so Ctrl-C works)
    while True:
            time.sleep(1)

This server operates as a simple polling loop over a client socket and the incoming audio data stream. Data received on the socket is placed in the write buffer used by the audio writer thread. Data received by the audio reader is send back to the client. This code could probably be cleaned up through the use of the select() call, but I frankly don't know if select() works with PyAudio and didn't investigate it. Given that the maximum data rate of the Superboard is 300 baud, a "good enough" solution seemed to be just that.

Putting it to the Test

Now, the ultimate test--does it actually work? To try it out, you first have to launch the above audio server process. For example:

bash % python audioserv.py
Reader starting
Writer starting
Server running on ('', 15000)

Next, make sure the Superboard II is plugged into the line-in and line-out ports on my Mac. On the Superboard, I had to manually type two POKE statements to make it send all output to the cassette output and to read all keyboard input from the cassette input.

POKE 517, 128
POKE 515, 128

Finally, use the telnet command to connect to the audio server.

bash $ telnet localhost 15000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
^]
telnet> mode character
LIST

OK
PRINT "HELLO WORLD"
HELLO WORLD

OK

Excellent! It seems to be working. It's a little hard to appreciate with just a screenshot. Therefore, you can check out the following movie that shows it all in action:

Again, it's important to emphasize that there is no other connection between the two machines other than a pair of audio cables.

That is all (for now)

Well, there you have it--using Python to implement a soft-modem that encodes/decodes cassette audio data in real-time, allowing me to remotely access my old Superboard using telnet. At last, I can write old Microsoft Basic 1.0 programs from the comfort of my Aeron chair and a 23-inch LCD display--and there's nothing old-school about that!

Hope you enjoyed this series of posts. Sadly, it's now time to get back to some "real work." Of course, if you'd like to see all of this in person, you should sign up for one of my Python courses.

posted by Dave Beazley  # 9:26 AM