Reading public filing data at scale: the EDGAR data tool

Every Monday morning a script I wrote pulls the previous week’s Form D notices from EDGAR, parses them, and drops a table on my desk. The table answers one question: what exempt offerings were filed last week, by whom, under which exemption, in which industry. The question is not new. What the tool changed is its price. A question that costs an afternoon gets asked once a year. A question that costs nothing gets asked every week, and a question asked every week starts to show patterns.

This article is about how the filing system is put together and how a small tool reads it. I wrote the tool for my own use and released it as open source. I have not linked it here because, at the time of writing, I could not locate a public repository for it under my account, and I would rather describe it than point at the wrong thing.

How EDGAR is organized

EDGAR is the Electronic Data Gathering, Analysis and Retrieval system, the public database of filings made with the Securities and Exchange Commission. Every filer has a Central Index Key, a numeric identifier that stays constant through name changes. Every filing has an accession number that identifies the submission, and every submission contains one or more documents: the primary document, exhibits, and for structured forms a machine readable data file.

The system publishes indexes. A daily index lists every filing accepted that day by form type, filer and accession number. Quarterly full indexes list everything in the quarter. There are also JSON endpoints that return a filer’s complete filing history and, for companies that report financial data in structured form, the data itself. All of it is public, all of it is free, and all of it is served under a fair access policy that requires a declared user agent identifying the requester and limits request rates.

Why Form D is the tractable case

Most filings on EDGAR are documents written for people: annual reports, prospectuses, current reports. Reading them at scale means parsing prose. Form D is different. It is a short structured notice filed after the first sale in an exempt offering, and since 2009 it has been filed as a machine readable data file with a fixed set of fields: the issuer’s name and address, its industry group, the exemption relied on, the date of first sale, the related persons, and a few figures about the offering that I do not use and do not reproduce.

Because the fields are fixed, the file can be parsed without any understanding of language. That makes Form D the easiest window into private capital activity that exists, and it is why the tool reads Form D rather than anything else.

What the tool does, step by step

The tool runs in four steps. First it reads the daily indexes for the period requested and keeps the rows whose form type is Form D or an amendment to one. Second it fetches each submission’s structured data file, respecting the rate limit and identifying itself in the user agent as the access policy requires. Third it parses the fields into a flat record per filing: identifier, issuer, state, industry group, exemption claimed, date of first sale, and whether the filing is new or an amendment. Fourth it writes those records to a table that can be filtered and sorted, and it keeps the raw files so that any record can be traced to its source.

Nothing in that pipeline is clever. The value is in the shape of the output. A week of filings becomes a sortable table rather than a list of links, and the question that used to take an afternoon takes a filter.

What the data can tell you

At the level of a single filing, Form D tells you that an issuer claimed an exemption and began selling. It tells you which exemption, which matters because the two halves of Rule 506 differ in whether the issuer may solicit publicly. It tells you the industry the issuer chose from a fixed list, its location, and who its officers and promoters are.

At the level of many filings, it tells you what is being raised where. Which industries are active in a state this quarter. Whether issuers in a sector are choosing the solicitation permitting half of the rule or the other. How many new filings appear against amendments, which is a rough read on new activity against continuing activity. For someone whose job includes competitive research across the private capital market, that is the map, updated weekly.

A worked example without the figures

Here is what a week’s table looks like in practice, with the figures removed as they are in the tool itself. A hundred or so new notices, and a smaller number of amendments to earlier ones. Sorted by state, a handful of states account for most of the rows, which is the pattern every week. Sorted by industry group, pooled investment funds dominate, because fund formation runs on the same exemption as operating company raises and there are many more funds than one might expect. Filter to operating companies in a single industry and a single state, and the list is short enough to read in a minute. Filter further to the half of the rule that permits general solicitation, and it is shorter still. That final list is the one I actually look at, because it is the set of issuers in my field who have chosen to raise in public, and their public materials will show how they describe themselves. None of that required anything more than a filter on a table that already existed. That is the entire argument for building the tool.

What the data cannot tell you

Form D is a notice, not a disclosure document. It does not say whether the offering succeeded, what the terms were beyond a few categories, who bought, or what the business does beyond an industry code. A filing can be made after the first sale and never amended, so a completed raise and an abandoned one can look the same. Some issuers file late and some do not file at all, so absence from the table proves nothing. And the figures the form does carry are self reported and are the fields I deliberately leave out of the table, because a figure without context is the thing most likely to be misread.

The tool therefore answers who, where, when and under what exemption. It does not answer how much, how well, or whether. Anyone who tells you a Form D scrape answers those questions has not read the form.

Building it responsibly

Three rules governed the build. The access policy is honored exactly: a real user agent with contact information, a request rate below the published limit, and no parallel scraping. The raw files are kept so that every derived record is traceable. And the table contains only fields that are public and structured, so that nothing in it is inferred.

The last rule is the one that took discipline. It is easy to join a filing to a company website, to a news article, to a person’s profile, and produce something that looks like intelligence. Each join adds an inference, and inferences compound. The tool stops at the filing.

The honest limit

Once a question costs nothing, it gets asked about things that do not deserve it, and a weekly table of filings can become a weekly habit of looking without a reason. The tool has value when there is a question in front of it, and none when there is not. The other limit is maintenance. The filing system’s formats change occasionally, the endpoints move, and a tool with one user has no one to report that it broke. I find out on a Monday when the table is empty.

Educational content only. Not legal, tax, or investment advice, and not an offer to sell or a solicitation of an offer to buy any security.