Extracting structured type information

6 min readOct 2, 2021

Using the typescript compiler api to infer types from javascript and typescript files.

If you are in a hurry, here is the results that you can expect:

And a link to the structured-types repository and documentation.

Genesis

One of the first features of component-controls was automatic documentation of react components.
However, react code isn’t just components — it’s also full of functions (such as utilities, common/repetitive code, etc). So, early last summer, I started work on automatically documenting generic typescript/javascript API functions.

For pure documentation of functions, most available libraries are doing fine, even though some of the detailed information is presented just as a string (like in this example)

However, component-controls needs detailed type information to automatically ‘execute’ components and API functions to automatically create tests. This requirement was the main reason for starting the structured-types project — to extract detailed, yet comprehensible type information.

Other libraries

The space for extracting documentation data from javascript/typescript files is already well served from a variety of libraries, where most of them support JSDoc-type documentation:

jsdoc — for documenting APIs written in javascript.

typedoc — for documenting APIs written in typescript.

ts-json-schema-generator — for documenting APIs written in typescript.

documentation.js — for documenting react and vue components written in javascript, flow.

react-docgen-typescript — for documenting react components written in typescriptc.

react-docgen — for documenting react components written in javascript, flow or even using react-proptyes. The latest version also handle typescript react components to some extent.

Development

Parsing

babel-ast
My first impulse was to use babel-parser,(which I was more familiar with and already supports typescript and flow) and then iterate through the ast-tree to retrieve symbols and their types, and for comment-type nodes use a separate JSDoc parser. This approach worked fine until trying to extract type information on class-like types that inherit another class-like type located in another file. Trying to “follow” all the inherited classes/interfaces and collect their properties was too complicated/hacky to consider.
typescript compiler api
At this point came to the rescue the typescript compiler api — it has api calls to get all the properties of a type, including inherited props, can infer types from javascript and typescript files, and even has a built-in JSDoc parser. Although the learning curve can be daunting at times (it is less documented than ast trees), it is well worth the effort if you need to extract type information.

2. Plugin architecture

Documenting components is a bit more involved than documenting regular functions or classes. The code declaration is pretty simple, but the interesting part goes through a series of indirections.
For example the following react component can display a configurable name property

import React from ‘react’;interface ComponentProps {
  name?: string;
}export const MyComponent: React.FunctionComponent<ComponentProps> = ({ name = ‘hello’ }) => <span>Hello, {name}!</span>;

The typings for React.FunctionComponent provide some utility properties, such as propTypes(which redirects to the ComponentProps property in the example above), contextTypes, defaultProps and displayName. For simplicity of reading the component’s props, we would want to transform the inferred type to a more readable type — with properties, and their values:

{ 
  “MyComponent”: { 
    “name”: “MyComponent”, 
    “extension”: “react”, 
    “kind”: 25, 
    “properties”: [ 
      { 
        “name”: “name”, 
        “kind”: 1, 
        “value”: “hello”,
        “optional”: true 
      } 
    ] 
  } 
}

The plugin architecture was designed to solve this kind of design issue, by allowing the transformation of the inferred component types.

Getting started

Install

$ npm install @structured-types/api — save-dev

2. Import into your code
structured-types exports two main entry-points
- parseFiles: where the parameters are a list of the files to analyze, the parsing options and some typescript compiler customizations.
- analyzeFiles: is the same as above but also will read the typescript configuration file associated with the parsed files.

import { parseFiles } from ‘@structured-types/api’;const docs = parseFiles([‘../src/sum.js’]);

3. An example
- Given the following javascript code, enhanced with JSDoc comments:

/**
* sum api function
* @remarks
* Unlike the summary, the remarks block may contain lengthy documentation content.
* The remarks should not restate information from the summary, since the summary section
* will always be displayed wherever the remarks section appears. Other sections
* (e.g. an `@example` block) will be shown after the remarks section.
*
* @param {number} a first parameter to add
* @param {number} b second parameter to add
* @returns {number} the sum of the two parameters
*
* @example
*
* ```js
* import { sum } from ‘./sum’;
*
* expect(sum(1, 2)).toMatchObject({ a: 1, b: 2, result: 3});
* ```
*/export const sum = (a, b = 1) => ({ a, b, result: a + b });

We will get the following JSON result

{
  “sum”: {
    “name”: “sum”,
    “kind”: 11,
    “parameters”: [
      {
        “kind”: 2,
        “name”: “a”,
        “description”: “first parameter to add”
      },
      {
        “kind”: 2,
        “name”: “b”,
        “value”: 1,
        “description”: “second parameter to add”
      }
    ],
    “examples”: [
      {
        “content”: “```js\nimport { sum } from ‘./sum’;\n\nexpect(sum(1, 2)).toMatchObject({ a: 1, b: 2, result: 3});\n```”
      }
    ],
    “returns”: {
      “description”: “the sum of the two parameters”,
      “kind”: 2
    },
    “tags”: [
      {
        “tag”: “remarks”,
        “content”: “Unlike the summary, the remarks block may contain lengthy documentation content.\nThe remarks should not restate information from the summary, since the summary section\nwill always be displayed wherever the remarks section appears. Other sections\n(e.g. an `@example` block) will be shown after the remarks section.”
      }
    ],
    “description”: “sum api function”
  }
}

4. Configuration options

tsOptions: ts.CompilerOptions — typescript compiler options.
internalTypes: Record<string, PropKind> -internal types - libs by default includes classes such as `String`,`Function`,...
extract: string[] — list of export names to be extracted. by default all exports are extracted.
filter: (prop*: PropType) => boolean — filter properties function. By default filter out all props with ignore === true.
isInternal: (file*: SourceFile, node*: Node) => boolean | undefined — callback function to determine if a node is an internal (typescript) symbol return undefined if you need to use the default isInternal processing.
maxDepth: number — max depth for extracting child props. default is 5.
collectHelpers: boolean — whether to save "helper" props that are used by the main parsed props if set to false will result in a smaller result set.
collectGenerics`: boolean- whether to collect generics parameters.
collectParameters`: boolean — whether to collect function parameters.
collectProperties: boolean — whether to collect object/type properties.
collectInheritance: boolean — whether to collect the inheritance properties.
collectExtension: boolean — whether to collect the plugin/extension name.
collectDiagnostics: boolean — whether to collect errors/diagnostics.
collectInternals: boolean — whether to collect internal (typescript) symbols.
plugins: ParsePlugin[] — installed plugins can modify default options and install type resolvers.
scope: "exports" | "all" — by default collects only the exported symbols.
collectFilePath: boolean — whether to collect the file path of objects.
collectLinesOfCode: boolean — whether to collect the source code.
location for the symbol declaration if set to true, the data will be collected in the loc prop .

Playground site

You can experiment with our javascript and typescript files in our playground site

Select examples
By clicking on the Examples tab, you can select various pre-created examples, such as JSDoc, Typescript, React and React-prop-types.

Analyze the results
The right-hand panel displays the JSON results of structured-types — you can expand/collapse the tree nodes and visualize the parsed types.
Compare documentation APIs
The playground also includes several external type parsers, that you can enable from the select viewers menu.
Change options
The parsing options can be changed from the Parse Config tab.

And the typescript compiler options from the TS Config tab.