description

Improve the jupyter publication pipeline experience; integrate cli functions into logger; a global, hierarchical storage container with reference semantics: metayer is an eclectic productivity toolbox intended to help get a new project up and running quickly.

description

The difficulty of generating simultaneous results for different runtime configurations was, ultimately, a shortcoming of DVC. The issue is that DVC understands DAG linkages in terms of input and output files, which are fixed. If a compute stage changes its runtime behavior and, hence, its output, then the simultaneous existence of both resultsets necessarily requires different file names.

I built an alternative to DVC that solved this problem. It required that stage dependencies be explicitly defined, and stage outputs be named according to a stage prefix and the hash of the upstream portion of the configuration. To accommodate resultsets comprised of heterogeneous data, I utilized the concept of a manifest, and it was this manifest that enjoyed the naming semantics while also providing an entry point / index to any items in the resultset. Much of this was handled seamlessly by the arrow R package.

Finally, I added an abstraction layer that allowed simple access to the upstream and downstream data. A participating function would call "load_result" at the beginning and "save_result" on termination. If upstream data was cached, that data would be loaded; if not, it would be generated from the upstream configuration. This meant that client code was stage-focused and relatively unburdened by any details of upstream data management.

description

I wanted to promote code modularization, and we investigated multiple, existing R packages that could have helped us better organize the code. None of them satisfied the principal investigators, so I made an attempt.

The idea was to expand R's notion of a package -- that is, source code in an ./R directory -- to a directory hierarchy. Using the R native ideas chained namespaces and dynamic scoping, each subdirectory provided scope for the code defined inside it. Further, these were chained up the tree, such that code in ancestor nodes was visible to, if not explicitly shaded by, descendents. This mimics object hierarchies in OOP languages admitting a rudimentary, module-level polymorphism.

Moreover, this can be combined with little additional effort to replicate traditional coding patterns like separation of interface and implementation. It also established a pathway to external runtime parameterization via a config file.

description

I built a general, fault-tolerant, high-throughput, data-retrieval framework that managed the requests to the POLIS API, logged and respawned timed-out queries, validated records, and stored a compressed version of the raw data. The raw data were augmented with additional metadata including time of retrieval, a hash identifier, and an indicator of whether the record validated against the POLIS-provided schema.

While it only collected data from four distinct POLIS tables, the system was designed to be general and extensible. In particular, it used OOP and a class registry paradigm to easily construct new data adaptors.

The file system was abstracted using the fsspec module, and both Azure and local storage options were made available.

Under the hood, I used the asyncio Python package to build an asynchronous work queue that managed concurrent HTTP requests.

description

I built an interactive webtool for visualizing how users traversed the instrumented state space. Using the D3 javascript library, I designed a circular Sankey diagram. This captured the assumption that users will likely return to various parts of an app during the course of a session. The tool also provided a selection widget, allowing states to be toggled on and off, with the idea that only certain states would be of interest.

For three different partners, I packaged state transition data into a preprocessed file that was consumed by the webtool. The tool was able to show that these state spaces were, in fact, quite large, on the order of hundreds of states.

With a little bit of markov chain math, it was relatively straightforward to pare down the state space to show (effective) transitions between states in the selected subset. Thus, users were able to dynamically target portions of the state space, obtain adjusted transition probabilities, and visualize the results in real time.

description

Specifically, the approach was to define a DAG from the pipeline stages using a parameter configuration file. Then, in cases where upstream portions of the parameter set had been previously computed, a cached result was supplied to the user. This provided consistency for data analysis and cost efficiencies because results weren't computed multiple times.

In addition, the library exposed an API for low-overhead, per-task logging through log4j / slf4j. Moreover, the library was designed to be useful across execution context: for local testing, deployed as a JAR in databricks, and as a databricks Job submission tool through the command line.

Abstracting the logging across context was intended to ease adoption amongst data scientists for the purposes of analysis instrumentation.

description

The RFP became a pretext for implementing a new and improved Stream ID product. The goals were to introduce a new, more flexible household model; to deliver an easily configurable, highly tunable, and interpretable model; to define a principled approach focused on time granularity and canonical graphical representations; and finally, to produce something that, if correctly parameterized, could recover the behavior in the existing implementation.

These concerns were guided by customer use cases.

In particular, it was important to be able to tune the model around some notion of time scale: should clusterings reflect yesterday's data, last week's data, or last month's data? Alternatively, perhaps the dynamics of the graph are of interest, and older data should be discounted.

It was also clear that customer context should inform whether Stream ID would work with clusters of clients, clusters of addresses, or clusters that include both types of nodes.

At a high level, it made sense to reframe the problem as a clustering on a weighted, colored graph.

But how to define weights? Again, customer context should inform the configurations that would be necessary. I termed these "Association Metrics", and provided implementations for session length (usage), session counts (incidence), time discounted session length and time discounted session counts. Each of these defines a weight between a client and an address. Such weighted, colored graphs can be reduced to single color subgraphs, with new weights induced by those in the larger graph.

Another metric, one that mimicked the existing approach and was applicable for client-client and address-address graphs, was based on the idea of overlapping intervals. Effectively, this approach considered each client-address edge and computed an interval from the first time the pair was observed to the last time the pair was observed. The weight between two clients (or two addresses) was the overlap of their intervals aggregated over all addresses in which the clients were co-incident. Time discounted versions of these metrics were also made available. During development, it was discovered that the overlapping intervals metric tended to create larger subgraphs, one of the most common complaints amongst customers.

Another concern was dealing with the competing objectives: is it more important to associate every entity (node) with a cluster or to produce smaller clusters? We addressed this with a posthoc pruning step.

One line of thinking that was in vogue at the time was that Conviva should purchase third party labels that classified IP addresses as residential or non-residential. That meant the v2 solution needed to provide a mechanism for a-priori filtering.

Postal codes were another feature of interest. Should they inform whether nodes could be members in the same cluster? So, I added a knock-in / knock-out feature which, again, could be toggled to fit the question of interest.

Another feature component evolved from a request to optionally map IPV6 addresses to a network prefix; and to optionally map client identifiers to a device fingerprint.

With edges now having semantic meaning, one defined by the choice of association metric, it made sense to enable thresholding for low value edges.

For the most part, the above amounted to easily configurable preprocessing steps; and the Louvain algorithm would be applied subsequently. However, the knock-in / knock-out feature required an additional blacklisting / whitelisting step inside of each Louvain iteration.

Finally, I instituted a postprocessing pruning step where vertices were removed from a cluster when their removal didn't significantly impact the modularity.

description

The problem was to compare Conviva's partitioning of IP addresses to a partition produced by an external, third party. Neither source has an exhaustive list of IP addresses, and, while most IP addresses will be present in both partitions, each partition will also contain IP addresses that are not available to the other.

The existing assessment metric classified each equivalence class (household) in the local partition. The classification scheme was based on how the local equivalence class was represented in the external partition; essentially, a covering. If there was a perfect match, it was labeled as one-to-one. If the local equivalence class was properly contained by an external equivalence class, many-to-one. If the local equivalence class was comprised of more than one external equivalence classes, one-to-many. Otherwise, many-to-many.

This worked on the subset of IP addresses that both partitions had in common, and the prevailing logic was that local equivalence classes that weren't in the common collection could be filtered out a priori. Equivalence classes that contained a mix of shared and unshared IP addresses were assumed to be negligible, and IP addresses known only to the external partition were deemed to be of no concern.

So as not to disturb intuition, one requirement was that the new metric not be too different from the existing one. One idea, then, was to construct a bipartite graph of local households and external households, introducing an edge between the two sets when there is a common IP address. Then, compute the connected components and report the cardinality of each side.

Thus, a component labeled as "1-3" would mean one local equivalence class is comprised of three external equivalence classes. In fact, this is a subclass of the one-to-many. A "2-2" would mean that the union of two local equivalence classes was equivalent to the union of two external equivalence classes, but in such a way that the corresponding subpartitions were not equivalent; this, a subclass of many-to-many. However, this approach also admits "0-i" and "j-0" subclasses, which better addresses the missing data concerns.

Another approach, one not taken here, might be to consider a Jaccard distance for each address that occurs in both datasets. Then, each household score would be a functional of the component address scores.

description

The goal was to stochastically capture "small-town" behavior.

I started with a fixed number of households, each associated with a randomly generated number of occupants. Occupants had a device which reset its device id at randomly generated intervals. Similarly, the household id was also reset.

Occupants were assumed to consume streaming video at a location, sporadically, throughout the course of their day. The onset and duration of these events was stochastic, as was the time between video sessions. However, an occupant could not consume multiple streams concurrently. This was a structural constraint of the simulation.

To incorporate the location, consumption could occur at home and also at an occupant's place of work. However, when an occupant left their house or workplace, they could choose to stop at one of several third places as well as other homes. This allowed more realistic mixing of devices and locations.

Another structural constraint in the model was that each occupant returned home at the end of the day, perhaps first watching more video, before starting a period of rest where no video was consumed.

Finally, I introduced time-of-day event rates that pushed people to be at work at a reasonable time as well as to leave at a reasonable time.

The generative model was, at a high level, a composition of random variables that characterized time-to-event and event-choice. Priors allowed occupants to behave with some additional variability.

description

I introduced back testing into an existing methodology which showed that 13% of presumed-to-be-required resources were never needed.

I built and developed a cohort- and market-segmented analysis using a markov chain model designed to capture invitation lag, onset spike, and steady state behaviours. This new model suggested that the existing methodology consistently over-estimated monthly steady state capacity needs by 100% to 300%.

I deployed calibrated models into Excel to enable real-time, what-if scenarios for the finance team. The excel add-in used the Python xlwings package.

description

Typically, this sort of question is best addressed with randomized controlled experiments. However, randomization requires advanced planning. So the best available course of action was a retrospective analysis.

For each user, I aggregated activity in the 28 days before receiving an xCloud invitation and activity in the subsequent 28 days. The user-level metric was then the difference between these two aggregates. This was identified as a pseudo-treatment group.

In order to make a comparison, we needed a pseudo-control group. This was problematic as there was no comparable subpopulation. Those participating in xCloud were chosen because they were some of the most active gamers. It also didn't make sense to compare those that chose to accept the invitation against the group of invitees that declined: that, too, was going to introduce bias.

Ultimately, we constructed a sample of users who had not been invited to participate. We introduced a change-of-measure / weighting scheme that rebalanced the subpopulations -- non-invitees, invited but declined, and invited and participated -- according to their behavior in the 28 days before invitations were sent out. This provided some comfort that the comparison might now be more meaningful.

The results of the analysis were inconclusive: there was no evidence to support the hypothesis that access to xCloud diminished or increased Xbox console usage. The flip side of this was that users weren't shifting their gaming minutes to xCloud. If anything, our post-mortem deep dives suggested that xCloud activity was largely a novelty effect.

description

A Bayesian hello world, I reproduced tables and figure in Chapter 3 and Chapter 4 of the Gelman and Hill 2007 text. There was unexpected data ingestion work, paring down lightly documented, CSV datasets for various tasks. It was also a first exposure to the Stan library.

Over the years, I've read Elements of Statistical Learning multiple times. One of those times, I worked through some notational stumbling blocks in one of the book's early chapters.

description

I wrote two python packages containing useful extensions and utility functions.

Metadata By Design

One of the common frustrations of data analysis is confronting poorly documented datasets. Categorical variables can be particularly painful. They are often coded and stored as integers, and without proper metadata or apriori domain knowledge, can easily be ingested as a covariate that wrongly enjoys the same properties as the natural numbers. This has the potential to silently break statistical and machine learning models.

This post introduces a pandas extension that more tightly couples categorical codings with their metadata. It guides users toward better data design.

Dynamic Query Construction

This module provides helper two objects, DatabaseManager and SqlQueryManager. The former provides a simple copyto method for copying pandas.DataFrames to PostgreSQL tables; and a more general, dbexecute method, for sending arbitrary SQL queries to a PostgreSQl server and returning any recordset provided as a pandas.DataFrame. All communication with the PostgreSQL server is via the python package, psycopg2.

SqlQueryManager relies on a DatabaseManager to provide a toolset for iterative, incremental SQL query construction. It effectively manages partial query dependencies via a dynamically constructed with_clause.

Jittered Histograms in Matplotlib

This module provides matplotlib helper functions for jittering data, adjusting colorbar height, and adding captions. There are also simple classes extending locator and formatter objects for clipped data.

description

The look and feel was inspired by the jsonresume.org "elegant" theme. The project depends heavily on the Python Klein library, Jinja2 templating engine, and pandoc to transform Markdown into HTML. The platform also permits easy inclusion of external, templated HTML; page-specific JavaScript; and registered python plugins associated with ajax endpoints on the server.

description

The featureset:

• full multivariate model enabling fast, online analysis
• native support for level, trend, and seasonality components
• non-informative priors for state space initial conditions
• automatically handles missing data
• support for modeling intervention effects

description

I built a bivariate Kalman filter with level, trend, and seasonality components and adapted its missing data functionality to monitor potentially anomalous deviations. The key idea was to introduce a second filter at the time of an outlier-sized jump and allow the second filter to evolve as though it was receiving missing data. If the two filters didn't resolve within some specified window, then the jump was determined to be real and the need for an alert could be assessed. Otherwise, the event was considered to have been spurious noise, and no alerts were issued.

description

This correlated event data problem was non-standard in the sense that event arrivals were potentially censored and could arrive out of order. So event A might cause event B, but due to logging and cache vagaries, it was possible to observe event B before event A.

The event domain was network resource log data, so we allowed that notable event pairs, were they to co-occur and to be observed, would do so within some fixed window of time. We then applied a sliding window to identify a candidate set of event pairs. Then, the problem to be solved was reduced to finding a subset of the candidate set where each element was likely to be a correlated event pair.

To accommodate the out-of-order scenario, we imagined that a correlated event pair necessarily shared a latent precursor event that induced the correlation. That is, the latent event would occur and each of event A and event B would happen subsequently with random waiting times. However, the observable data wasn't the usual waiting times; rather, it was the difference of the two times at which event A and event B occurred. This was enough to write down a likelihood function for each candidate event pair and recover maximum likelihood estimates of the branching parameter process.

I used the likelihood function evaluated at the MLE as a test statistic and compared it to a null hypothesis where each event type represented in the candidate set of event pairings was a realization of an independent Poisson process. Then, I applied the same machinery as described above to a simulated dataset from this null model, again computing the MLE as a test statistic. Repeating this process, across a collection of simulated datasets, produces a distribution of test statistics under the null at which point a hypothesis test (or p-value) is readily available.

Because the test statistic is evaluated at the MLE, asymptotics would suggest that approximation approaches might be available. However, this flips the rolls of what we've identified as the true and null models. That is, the approximation should be taken about the MLE from the true model, and so the null model provides a test statistic which is now the "unusual" observation. This asymptotic approximation reduces the computational requirements dramatically. As might be expected, it was also found to be most reliable when observed event pair counts were high.

description

I wrote a C# library of custom SCOPE mappers / reducers to compress millions of records into quantile lookup tables. These were suitable for visualization (reinterpolated histograms) as well as for computing more general statistical functionals. After compression, these latter analysis tasks were easily handled with locally running code.

description

I was tasked to work with the OneNote and Exchange Online teams. We initially worked with PMs to identify usage metrics that could be aggregated into an overall evaluation criteria. Then we engaged with the engineers to ensure that products were instrumented and that data collection was up, running, and of high quality.

description

Teams within the Bing umbrella would typically launch their experiments in a self-service capacity, and this was the same experience we planned to offer our partner teams. However, there was an unpublished, internal command line tool that could also be used to launch experiments.

I built a Ruby on Rails app that provided a simple web interface to that command line tool. The user would provide an identifier for their experiment. They also needed to provide a SCOPE script that filtered their customers to experimental subjects and computed the metrics they wanted to monitor. However, for partner teams, this was readily available: it was a checkpoint on our engagement roadmap. This script only needed to be mapped into a Bing compatible format. The app provided this adaptor, building a dynamic SCOPE function that could be ingested into the command line tool.

description

The request was to produce a five year forecast of active users from the previous three months of data. What do you do in that situation? Upfront, I explained that there was no way to provide a meaningful answer, but was told that it had to be done. So I built a simple time series model and, as expected, it produced an absurd result: with 95% confidence, either there'd be no users or everyone on the planet would be on O365.

For the PMs that had demanded this work be done, it wasn't a satisfying answer. They felt the request hadn't been taken seriously. As a response, I scheduled a follow up where I was able to communicate some of the basic limitations of statistical models. At that meeting, I was also able to point out where irregularities in our dataset contributed to even wider confidence intervals. If we hoped to improve on the initial results, the model would need more context.

That was the a-ha moment. The PMs recognized that one of these unusual periods of growth corresponded with a known event: a developer had pushed a change that introduced more data into the system. This had happened on a ramp, so it wasn't immediately identifiable as a jump -- the model saw it as evidence that a twenty percent increase in weekly usage wasn't implausible. What was interesting was that far more of the irregularities that we reviewed couldn't be accounted for. This was an indicator that we needed to make investments in tracking this sort of metadata.

At that time, there was no channel or process for communicating changes in the behavior of the data pipeline. Rather, these sorts of events were inquired about after the fact, usually because a KPI changed without any other explanation.

My last project with the Customer Intelligence team was to build a change point model, so that PMs could approach engineering to inquire if something had, indeed, changed.

description

The team leadership had been working under the false impression that the volume of support tickets would be a fixed cost and not one that scaled with the size of the userbase or the level of user activity. This was a strongly held belief, and there was resistance to the idea that the data might provide evidence to the contrary. The logic, as I understood it, was that if the fixed cost hypothesis was incorrect, then the business was doomed. Existentially, it needed to be something else.

One of these alternative hypotheses was that customer support simply wasn't doing a good job, and my team allocated a two week block where everyone was asked to listen in to customer support calls and manually map the concerns into a fixed, knowledge base taxonomy. The idea was to compare our manual labelling of issues with those assigned to the ticket by customer support. While the hope was to observe a failing of customer support, the two collections of labels were found to be largely in agreement.

My contribution was to do the analysis that nobody wanted: to decisively show that more customers is correlated with more problems. We were going to have to face a tough truth.

This wasn't the only anti-pattern on the team. Another failure mode was reactive reporting. Every week, the team would produce a list of worst offenders; those ticket classes that dominated the call centers. Then, there'd be an ensuing scramble to find an explanation. As a statistician, it's second nature to ask how much of a thing is simply natural variation. So I attempted to frame the problem in those terms.

In conversations with PMs, two scenarios emerged as candidates for explaining week over week natural variation. The first was that tickets could be mislabeled, even if infrequently. The second was related to intermittent data loss. Both of these would contribute to week over week variation.

I built a stochastic model for each of these scenarios to assess variances. They were conservatively parameterized using domain knowledge, and the results strongly suggested that in most of our weekly reporting postmortems, we'd been out on fishing expeditions.

description

The team was given a taxonomy for customer support tickets, and leadership wanted to know if any of the ticket types were correlated with onboarding issues or other early lifecycle concerns. One simple way to address the question is, for each ticket created, to consider the time elapsed since account creation (or, perhaps, first use). Then each leaf node in the taxonomy can be interpreted as a sample from a distribution of a random variables characterizing the elapsed time. The collection of distributions (across nodes in the tree) could then be characterized with a Bayesian prior. A principled approach would incorporate the hierarchical structure, perhaps by treating each internal node of the taxonomy as a mixture of the distributions of its children.

My manager's response to this analysis proposal was profound confusion. The proposal might as well have been communicated in a completely different language. Over the course of that conversation, it became clear that he needed a fast turnaround far more than he needed statistical sophistication.
The new plan, then, was, for each ticket class we'd compute a mean along with student-t confidence intervals and report those that weren't significantly larger than a relevant threshold, a two weeks onboarding window. We talked about the risk of false positives (multiple testing problem), and he communicated that he was willing to ignore that risk.

description

A secondary page on the site fused geographic (TIGER/Line) and census (ACS,BLS) datasets. This featured an interactive heatmap that reported spatial statistics aggregated at a metro (MSA) level. While simplistic today, in 2013, interactive maps weren't quite as ubiquitous.

description

One facet of this analysis was assessing the survival risk of a given loan and using this to inform a continuous reinvesting strategy. This helped guide the choice of objective function in the underlying optimization.

The primary contribution was adapting a random forest to this survival paradigm.

Our results, driven by expected returns, compared favorably to investment strategies based solely on the loan grade assigned by Lending Club. Our optimal, actively managed portfolios had an expected return exceeding 12% annually. In contrast, portfolios constructed on A-grade loans return 6.68%; B-grade loans, 7.49%; and C-grade loans, 8.11%.

However, as an investment vehicle, the primary concern was the availability of "good" loans as it was well known that investors had early access to available assets.

description

I used an Apriori algorithm to create new features from historical purchases. These attributes had higher predictive power and produced significant lift in our models.

I provided statistical support for implementing a step-wise logistic regression model that was put into production.

description

Microsatellites make up about three percent of the human genome, and there is increasing evidence that some microsatellites can have important functions and can be conserved by selection. To investigate this conservation, we performed a genome-wide analysis of human microsatellites and measured their conservation using a binary character birth-death model on a mammalian phylogeny. Using a maximum likelihood method to estimate birth and death rates for different types of microsatellites, we show that the rates at which microsatellites are gained and lost in mammals depend on their sequence composition, length, and position in the genome. Additionally, we use a mixture model to account for unequal death rates among microsatellites across the human genome. We use this model to assign a probability-based conservation score to each microsatellite. We found that microsatellites near the transcription start sites of genes are often highly conserved, and that distance from a microsatellite to the nearest transcription start site is a good predictor of the microsatellite conservation score. An analysis of gene ontology terms for genes that contain microsatellites near their transcription start site reveals that regulatory genes involved in growth and development are highly enriched with conserved microsatellites.