This is a short write up on how I am getting live data from the New York city subway https://new.mta.info/ (metro, underground, U-Bahn, however you call it) in TouchDesigner https://derivative.ca/. It is a bit terse and requires some tinkering, installing, using git and changing settings. Hit me up on Twitter if you have any questions (https://twitter.com/crcdng).
Many live data feeds come with a REST API where you can call a web service and get data back back in JSON format. The realtime subway data uses a different mechanism called GTFS Realtime, which is based on Protocol Buffers (https://developers.google.com/transit/gtfs-realtime/). I believe you can get the data in JSON format for Google Maps, but here I will retrieve it directly with Python, the included scripting language of choice in TouchDesigner.
Before retrieving the data within Touch, I will get it with a Python script running outside of Touch. The reasons are (1) in this way the necessary python libraries can be installed, (2) the script can be tested (3) if you want to keep things modular you can use the script instead of steps 7 and 8 below as a base to send data into Touchdesgner, e.g. with OSC, and finally (4) someone has already done it 😀👏👍.
To use the subway live data, get an MTA API key. It is free but requires to sign up here: https://datamine.mta.info/. The key comes by mail, e-mail.
If you haven't installed Python 3, do it now. It needs to be the same version as TouchDesigner's built-in Python which as I write this is 3.7. I heavily recommend to always install Python with a virtual environment such as venv or Miniconda (or maybe a future one that always works).
To install Miniconda (64 bit), create a Python 3.7 environment and switch into this environment, follow the instructions here: https://docs.conda.io/en/latest/miniconda.html. These steps involve using a Terminal on a Mac / Linux machine or a CMD Shell on Windows. Keep this window open.
Make a note of your module directory path. I have installed Miniconda on my D: drive on Windows 10 where my path is: D:\<MY USERNAME>\miniconda3\envs\<MY ENVIRONMENT NAME>\Lib\site-packages. On my Mac, it is /Users/<MY USERNAME>/miniconda3/envs/<MY ENVIRONMENT NAME>/lib/python3.7/site-packages.
Your path will depend on the location of your Python and virtual environment installations and the name of your environment.
Thankfully, Bennet Garner has done the work and created a Python script to get MTA realtime data with Python. His write up is here: https://medium.com/@BennettGarner/parsing-gtfs-format-transit-data-in-real-time-with-python-3a528ba7aab7
In your Python environment (step 2), clone his Git repository https://github.com/bennett39/mta-real-time-data and follow the instructions to install the requirements (which will be used for the script in Touch as well).
You could put the MTA API key (Step 1) into your TouchDesigner project in a Text DAT but the recommended way is to read things like passwords or access keys from an environment variable. This works differently on Mac and Windows. On Windows 10, open Control Panel\System and Security\System and select Advanced System Settings. There, select Àdvanced > Environment Variables... > User variables for <YOUR USERNAME>. Click New and add the MTA API key (step 1) into a new environment variable called MTA_API_KEY. On Mac / Linux create a file called .env instead and put your MTA API key (step 1) into it like this (omitting the angle brackets).
API_KEY=<YOUR MTA API KEY>
Notice where and how your key is stored. if you share your project or plan to run it on another machine, depending on your circumstances.
Now run python gtfs.py in your terminal window. If everything is correct, you will find a file called output.txt that contains current MTA live data. Congrats, important steps are done. Two more to go.
If you want to keep things together or maybe run the project on another computer later, copy the complete site-packages directory from Step 4 into your project directory.
In TouchDesigner, go to the Preferences and make sure Add External Python to Search Path is checked. Enter the path to your site-packages directory (either the one from Step 4 or the copied one, if you copied it) into the field Python 64 bit module path. Save the change. Restart Touch.
More Information about Python and external modules is in section "Importing Modules" on https://derivative.ca/UserGuide/Introduction_to_Python_Tutorial.
To run a Python script inside of Touch, you have a number of options, e.g. a Text DAT, a Script DAT (comes with a Text DAT) or a DAT Execute DAT. You can trigger it manually, based on some other data, triggered by an event or you can call it periodically.
Here I am using a Text DAT that just prints the data.
Open Dialogs > Textport and DATs to see the data coming in.
Put the following code into a Text DAT and right-click > Run Script.
def get_feed(feed, url, completeP):
response = requests.get(url, allow_redirects=True)
feed.ParseFromString(response.content)
if (completeP):
# the complete feed
return (feed)
else:
# just the updates
return "".join([str(entity.trip_update) for entity in feed.entity if entity.HasField('trip_update')])
def main():
env_path = Path('.') / '.env'
load_dotenv(verbose=True, dotenv_path=env_path)
apiKey = os.getenv("API_KEY")
feed = gtfs_realtime_pb2.FeedMessage()
try:
url = "http://datamine.mta.info/mta_esi.php?key=" + apiKey + "&feed_id=1"
except:
# mta_data = op("example_data").text
# print(mta_data)
# print('MTA API key not found, this is example data')
print('MTA API key not found')
else:
try:
mta_data = get_feed(feed, url, True) # get the complete feed
except:
# mta_data = op("example_data").text
# print(mta_data)
# print('MTA API call not successful, this is example data')
print('MTA API call not successful')
else:
print(mta_data)It prints realtime MTA data into your Textport window, either the complete feed or just the updates. You can read more about the data format and adapt the functions to your ideas here: https://datamine.mta.info/feed-documentation.
Now what should happen if an error occurs, e.g. the key is not found or the internet connection is broken? I am using another Text DAT with example data as a fallback (see commented out lines 73 to 75 and 81 to 83). It really depends on your project how you want to handle missing data.
If everything works fine, you can now process the data further, for example write it into a Table DAT and go from there.
Have fun.