Our team’s composition and mandate influence our stack

The requirements for our data stack are downstream of our team’s composition and mandate:

• Because our team is small, our infrastructure can be lean, since an outage will only affect a limited number of users.

• Because our team is cross-functional, our infrastructure needs to be easy to maintain with minimal effort by team members who have varying degrees of familiarity with DevOps and MLOps.

• Because our team is technical, we want to be able to keep as much of our work as possible under version control; we want to automate the boring stuff, like continuous integration and deployment (CI/CD); and we want programmatic access to our data so that we can use it in scripts and apps.

• Because our team is autonomous, we want as much control over the design and operation of our data systems as possible.

• Because our core mandate revolves around data science, engineering, and analytics, we sometimes need access to larger compute environments in order to run batch jobs, and we want columnar data storage to make aggregate queries more performant.

It’s also important to note that the universe of our data is medium-sized. The basic unit of our work is the parcel, which is a unit of taxable real property that refers to a home, a commercial space, or vacant land. Cook County is home to roughly 1.9 million parcels, and since most parcel attributes are recorded once per year, our database tables tend to have record counts in the tens-to-hundreds of millions. That’s enough data that it’s often difficult to analyze on a laptop, but it’s much smaller than the data that most large private-sector data teams maintain, particularly teams that maintain real-time or event-driven data.

If your team doesn’t look like ours, or if your data look different from ours, then your stack will probably be different, too. Regardless, if you see some similarities between your team and ours, then we hope you’ll get some value out of understanding what we’ve built.

On that note, let’s dive into the stack.

Source-of-truth vendor database: iasWorld

The CCAO system of record is a database called iasWorld. A private contractor (Tyler Technologies) maintains this database along with an associated web app that allows CCAO staff to edit data in the system.

Beyond editing data, the iasWorld web app allows CCAO staff to perform core office functions like reviewing building permits and adjusting assessments based on appeals. While we often describe iasWorld as a database, since that’s mostly how the Data team interacts with it, iasWorld is actually a cloud Enterprise Resource Planning (ERP) system that encapsulates all of the essential business logic of the CCAO, from certifying assessments to adjudicating appeals. The iasWorld database and web app are also shared among the other Cook County offices that are part of the property tax system, including the Clerk, the Treasurer, and the Board of Review.

While iasWorld is the system of record for the Data team, there are a few reasons why the Data team does not use it directly for our day-to-day analytics and data science workloads:

• The iasWorld database is a row-wise, relational Oracle database, so aggregate queries are slow.
• There is often heavy load on the iasWorld database during the workday, so our query volume could overload the system.
• The iasWorld database has a basic query interface, but it does not support the programmatic SQL queries that we need for analytics and data science.
• We prefer to work with views that clean iasWorld’s raw tables and join them to other data sources, but our team does not have permission to create views in iasWorld, let alone introduce external data into the system.

These limitations of iasWorld are understandable, since the database wasn’t designed to support the types of data science and analytics workloads that our team runs every day. In order to enable those workloads, we need a read-only mirror that can handle a high volume of aggregate, programmatic queries, and we need it to support custom views that we can test and update as underlying data change. We also need to combine the data in our system of record with other public and private data sources in order to properly model parcel characteristics and report on the performance of our assessments.

To meet all of these requirements, we built ourselves a data lake.

Data lake: AWS Athena (S3 + Glue)

Our AWS Athena data lake is the foundation of the day-to-day work of the Data team. The data lake stores all of the raw, intermediate, and final tables and views that allow us to integrate public data sources with data from our system of record. It also allows us to query our data programmatically and at scale, which in turn enables us to deliver a degree of analysis and automation that would be otherwise impossible.

Athena is an Amazon Web Services (AWS) product that provides a serverless SQL query engine using AWS S3 for file storage and AWS Glue for schema management. To query some data in Athena, we store structured data files in S3, usually in the compressed Parquet file format, and then we configure Glue to read those files as tables. Athena then provides an interface for querying our Glue tables using SQL, and it also allows us to store common SQL transformations as views.

There are a number of advantages to using a serverless query engine like Athena for our data lake:

• Low cost: Athena charges us per gigabyte of memory that it scans during query execution, and S3 charges per gigabyte of files stored per month, so we only pay for what we use. Costs per gigabyte are generally quite low for both services, which means we almost never have to worry about exceeding our budget.
• Fast aggregation: Athena is able to read from columnar data sources like Parquet, allowing it to execute aggregate queries with greater efficiency than a row-wise database. This is helpful when running analytics and data science workflows, which often rely on aggregate queries.
• Horizontal scaling: Athena queries are parallelizable, and AWS manages the underlying infrastructure, so the query engine’s compute resources scale automatically to support our query load. In practice, this means that we never need to worry about accidentally slowing down our database with expensive queries.
• No maintenance: Since Athena is serverless, there is no underlying infrastructure that we have to manage. This frees up our staff to work on more important problems, and it means that our team doesn’t need an experienced database administrator in order to do our work.
• Programmatic access: Like all AWS services, Athena exposes a web API, and there are many well-maintained integrations that make it easy to use the API in different contexts. We rely heavily on Athena’s Python and R integrations.

Again, these advantages are downstream of our team’s composition and mandate. If our data lake had more than a few users, or if our data were larger, then we might end up spending so much on storage and query execution that we would benefit from moving our infrastructure in-house and hiring database admins to maintain it. Similarly, if our team were less technical, then programmatic access to our data lake would be less important.

Athena is the foundation for the data science and analytics work that we do every day, but it’s not useful without data. To get data into our data lake, we need to extract it from external data sources and load it into the data lake.

Extraction/loading (“E/L”): R scripts and Spark

There are two main ways that we extract and load data into our data lake:

1. Spark ingest for frequent iasWorld extraction: Since we need to extract a medium-sized amount of data from iasWorld and load it into our data lake on a daily basis, we use a Spark cluster to speed up the extract/load process for iasWorld data. We maintain this Spark code in a dedicated repo called service-spark-iasworld and we run it on an on-premises Ubuntu server using a Docker Compose service that we schedule using cron.
2. R scripts for infrequent external data extraction: In contrast to iasWorld, which other teams update with new data every day, most of the external data sources that we use are small and infrequently-updated. For example, we use Census data to derive neighborhood characteristics that we feed into our valuation models, and the Census only publishes data once per year, at resolutions that are less granular than parcels (we typically use tract-level Census data). Since these data are small, and since we ingest them infrequently, it’s more important that their extraction/loading processes are easy to maintain than that they are maximally efficient, so we define them as R scripts and we run them manually when needed. We store these R scripts in the etl/ subdirectory of our data-architecture repo.

Regardless of which extraction/loading process we use for a particular data source, the result is always the same: A parquet file stored in S3. Once that raw data is in S3, it’s ready to be cleaned up and combined with other data sources.

Transformation (“T”): dbt

Like most data teams, we maintain a large number of SQL transformations that clean our raw data tables, join them together, and make them easier to query. To manage these transformations, we use a tool called dbt.

To motivate our use of dbt, let’s look at a simple example: our modeling views. Our residential model and our condo model both need access to roughly 100 different parcel attributes (“features”) in order to accurately predict sale prices in the Cook County real estate market. Many features (like building size and age) are stored in iasWorld, but many others (like distance to parks and transit stops) come from external data sources; even the features that come from iasWorld may be stored in disparate tables. When we train our valuation models and use them to predict sale prices, we need to pull all of these data together, which requires a hefty SQL query.

In theory, we could write our modeling queries directly in our modeling code, but then we wouldn’t be able to reuse those queries anywhere else without copying them over entirely. We would prefer to have dedicated views in our data lake that encapsulate these queries so that we can test them and reuse them.

Athena allows us to save our modeling queries as views, and dbt allows us to create and edit those views automatically. Using dbt, we can write a SQL query for our residential model input view and save it to a file in our dbt project; then, we can define some attributes of that view in a YAML config file and use the dbt build command to create an Athena view based on the SQL query and its YAML attributes. With dbt, we don’t have to push files to S3 or create and run Glue crawlers, since dbt automates those steps for us. Instead, the view is entirely defined by regular files, and we can track those files in our version control system (GitHub) and set up an automated job to update the view whenever its associated files change.

Beyond managing transformations, dbt also has some convenient features for testing and documenting our data. We use dbt to define and run data tests that confirm basic assumptions about our data, and we also use it to autogenerate a data catalog based on the YAML config files that define our tables and views.

Another way of thinking about it is that dbt applies the principle of infrastructure-as-code to the data in our data lake. Using dbt, we can document the relations in our data lake using version-controlled queries and metadata files; we can automatically update those relations whenever their definitions change; and we can test and document our relations. This framework allows us to make a high volume of changes to a large number of transformations while having confidence that all of those changes will be reviewed, applied, and recorded correctly in a central source of truth (namely, GitHub).

Though the company that maintains dbt offers a paid service for this type of automated workflow, we use the free and open source version of dbt because we have the technical capacity to maintain our own automations. While this means that we don’t need to pay for dbt, it also means that we need an automation and orchestration layer in order to put it into production.

Automation and orchestration: GitHub Actions, plus VMs

The GitHub Actions platform allows us to run dbt whenever we push changes to the metadata files that define the transformed tables and views in our data lake. More broadly, it provides a general-purpose remote compute environment that we use for a variety of small automated tasks.

The basic unit of work for GitHub Actions is the workflow, which is a YAML config file defining a sequence of scripts that run on particular triggers, called events. The most common events that trigger our workflows include:

• schedule: Run the workflow at specific times based on a cron expression.
• push / pull_request: Run the workflow when a git event (commit to a branch or update to a pull request) occurs in the repo where the workflow is defined.
• workflow_dispatch: Run the workflow on demand, with an optional set of input variables that the workflow caller can specify.

When an event occurs that a workflow is configured to listen for, GitHub will spin up a temporary compute environment to run the scripts defined in that workflow. In this way, GitHub Actions is both an automation layer and an orchestration layer: “Automation” in that it can run tasks in a remote compute environment without requiring us to run commands directly, and “orchestration” in that it can trigger sequences of tasks that should run at specific times, or when certain prerequisite tasks execute successfully.

Data science workloads: AWS Batch + EC2

We use AWS Batch as an abstraction layer on top of AWS EC2 in order to run heavy data science workloads like our residential and condo valuation models.

AWS EC2 is a service that provides on-demand rental of remote compute environments, priced per second of usage. AWS Batch is a service that builds on top of EC2, providing a convenient interface for running batch compute jobs on ephemeral EC2 machines. To define a job in AWS Batch, you provide a Docker image for the job and you specify the amount of resources (CPU and memory) that the job requires; then when you’re ready to trigger the job, Batch will provision an EC2 instance that meets your resource requirements, execute the job on that instance using the Docker image you provided, and then tear down the EC2 instance when the job completes (or fails).

In order to trigger Batch jobs on demand, we use a GitHub workflow called build-and-run-model that listens for a workflow_dispatch event. The workflow creates or updates a Batch job configuration based on a set of input variables that the workflow caller specifies, then it starts a new job, printing a link to job logs that allow us to monitor execution. Using this workflow, we can trigger an unlimited number of model runs with the click of a button on a dropdown menu in the GitHub Actions interface.

Batch and EC2 are good fits for our data science workloads because they give us maximum flexibility with minimal configuration: We can use any of the hundreds of EC2 instance types as the basis for our compute environments; we can configure those environments with all necessary dependencies using a Dockerfile; and we can let Batch take care of setting up and tearing down the compute environment whenever we run a job.

Batch also allows us to easily hook our jobs into other AWS services, which comes in handy for logging and alerting.

Logging and alerting: CloudWatch

We use AWS CloudWatch as a remote log sink for all of our jobs and services. CloudWatch has a huge number of features, but we mostly use it for storing logs in a central location where they are easy to query programmatically.

These logs are useful for monitoring and debugging services, but they are also useful as a source of information about the status of our scheduled jobs. Since our orchestration layer is so simple (as we discussed in a previous section), we don’t have centralized job status reporting; instead, we maintain a repo called service-alerts with a scheduled workflow that runs every three hours and queries the CloudWatch logs for our services to make sure they produced the expected logs without running into errors. This system requires that we maintain custom code in order to run healthchecks against our services, which could be a long-term maintenance risk, but the CloudWatch API mitigates some of that risk by making the code easy to write and maintain.

Infrastructure management: Terraform

Since our stack relies on many AWS resources, we use Terraform to save the state of our AWS infrastructure in version-controlled config files. To do this, we maintain a private repo in which we store Terraform config files containing definitions for all of our resources, excluding IAM user accounts and their attached policies and groups.

While it’s common to use Terraform to automate the provision of AWS resources, we mostly avoid that approach, since it would require giving automated jobs permission to create and destroy many types of resources in our AWS account. This type of automation would increase the risk that the job might leak powerful credentials or accidentally incur downtime by deleting resources. Instead, we make most of our changes directly in the AWS console, and then we update our Terraform config after the fact to make sure it is up-to-date with the changes.²

Analytics workloads: Tableau

Tableau is our preferred data analytics platform. We connect Tableau to our Athena data lake in order to pull data into dashboards, and we use those dashboards as the basis for our internal and external reporting.

Some examples of public-facing Tableau dashboards that pull from the data lake include the Cook County Housing Market Tracker and our Historical Analysis of Property Tax Spikes. For a complete list of all of our public Tableau dashboards, see the Data Dashboards page on the CCAO website.

Tableau is a good fit for interactive data products, but it’s more limited in its ability to deliver raw data. We sometimes use it to deliver raw data to internal stakeholders, where we have some ability to train other teams to use it correctly, but we don’t rely on it for delivering raw data to the public. In order to give the public direct access to as much of our raw data as possible, we need a dedicated open data portal.

Open data: Tyler Data & Insights (FKA Socrata)

We use the Tyler Data & Insights platform, formerly known as Socrata, to host our open data portal.

In order to push data to the portal, we schedule a GitHub workflow called upload-open-data-assets to run twice a month. The workflow runs a Python script that extracts metadata about our open data tables from our dbt config, queries the relevant tables in our data lake, and uses the SODA Producer API to push new data to the portal. To make sure the upload works properly each month, we schedule a separate workflow called test-open-data-assets to compare data in our data lake to data in the portal, ensuring that they match. This workflow sends us an alert if the assets in our data portal don’t have similar row counts to the underlying tables in our data lake, which can indicate that the upload failed to complete.

The open data portal is the outermost layer of our data stack, pulling data from our data lake and delivering it directly to the public. See this guide for an overview of all of the assets that we publish on our open data portal.

Our orchestration layer will not scale

There’s a big problem with our orchestrator: Namely, that we don’t really have one. To see why this is a problem, let’s think about the jobs we run.

Most of our scheduled jobs depend on the daily Spark job that ingests data from iasWorld, our source-of-truth database. Currently, we configure all of our automated jobs to run on independent schedules, and we assume that upstream jobs like the Spark ingest will complete within a certain window of time.

This system is brittle, and it breaks easily. If upstream jobs fail, or if they run slower than expected, then the schedules for any downstream jobs will no longer be valid, causing our data products to display stale data. What’s worse, this type of failure can happen silently, since downstream jobs can often run successfully using old data.

For most of the history of the CCAO Data team, it made sense to stick with a simple-but-brittle orchestration layer, since our team maintained very few scheduled jobs that depended on the state of other jobs. Orchestration layers are notoriously complex systems that can be difficult to manage, and we preferred to save our capacity for higher-priority tasks.

However, as time has passed and our team has built trust with other departments, we find ourselves maintaining more and more automated data products that other teams rely on. As a result, we are starting to experience the costs of our lack of orchestration in the form of decreased reliability of those data products. Eventually, we will likely need to accept the additional maintenance burden of a sophisticated orchestrator like Airflow in order to continue expanding our suite of data products.

Automated jobs are scattered across compute environments

As we mentioned above in the section on automation and orchestration, we run automated jobs in a few different compute environments. GitHub Actions is our preferred compute environment, but we have to run some jobs on our Windows VMs and our Ubuntu server in order to access private data. We also run data science jobs on AWS Batch due to our historical reliance on the free tier of GitHub Actions, which didn’t provide us access to larger runners.

In the short term, these scattered compute environments don’t pose a major problem, though it’s often annoying to deploy code to our Windows VMs and our Ubuntu server given that those platforms don’t allow for automated CI/CD. However, these scattered compute environments are a long-term maintenance risk: The more compute environments we have, the harder it will be for the team to keep track of all of the jobs we’re running.

Eventually, we would like to migrate all of our jobs over to GitHub Actions so that they all run in the same environment. We could do this with a combination of self-hosted runners and larger runners. This migration would involve trading some staff time for the benefit of centralization, since self-hosted runners will require us to manage our own infrastructure rather than leave it up to GitHub.