MkTechDocs

A framework for collating and transforming markdown

1 Introduction

1.1 What is it?

MkTechDocs is an open framework for collating and transforming markdown. You can use it to manage complex technical documentation projects with dependencies, produce light documentation, or combine and transform existing Markdown files into a styled webpage or PDF document.

1.2 Features

• 100% open source
• No proprietary, hard-to-read, or hard-to-learn input formats or configuration syntaxes
• Standardized to Markdown with Pandoc extensions, which is easy to learn and convertible into many other formats
• Shallow learning curve
• Builds CSS-styled HTML, PDF, Microsoft Word, and markdown documents from the same source documentation
• Encourages modular document design
• Flexible and extensible
• Projects ideally suited for storing in git repositories, which means documentation work can be distributed like source code
• Python and Groovy/Java hooks provide easy access to pre-existing code libraries
• Jinja2 and Groovy template support is baked in for more complex documentation needs
• Pandodc templates can be customized on a per-project basis
• Agile and continuous-integration friendly

1.3 Can I see an example?

You're looking at one!

1.4 Supported architectures

MkTechDocs was built in and is most frequently used in Ubuntu Linux 18.04 LTS; however, MkTechDocs should work in any environment that supports these dependencies.

1.5 Install

See the Install MkTechDocs and Setting up your environment sections at the end of this document for more information.

2 The basics

The easiest way to learn how to use MkTechDocs effectively is with a tutorial project. Here's one on GitHub.

Unlike the comprehensive usage guide, the tutorial project uses a Docker image to build and provides a good overview of a MkTechDocs project without getting too far into the weeds.

2.1 Markdown

MkTechDocs uses Markdown exclusively, although support for additional syntaxes may be added in the future. More specifically, it uses standard Markdown syntax with Pandoc extensions. In this documentation, assume that any file with a .md extension is a file that contains Markdown. Markdown is easy to learn, read, and edit and is easily transformable into a myriad of output formats. To use MkTechDocs effectively, take a few minutes to familiarize yourself with Markdown syntax.

2.2 Creating a new project

To create a new MkTechDocs project, cd inside the directory where you want to create it and run mktechdocs. For example:

mkdir mynewproject
cd mynewproject
mktechdocs init

mktechdocs will create the following files in mynewproject:

* NOTE 1: Optional. If you don't need this HTML content, for example, if you're producing PDF content only, you can safely delete these files.

* NOTE 2: If you need more dynamic content in your header, footer, or main content, dates, database information, etc., please read the footer.html, header.html, and landing.html as templates section

2.3 MkTechDocs projects

2.3.1 Master document

MkTechDocs projects consist of a "master" document — a single markdown document that defines your project, always named master.md— and any number of sub documents. Building a master document is simple:

```comment
Define one or more include blocks to include all the major subdocuments that
make up your project.
```
```{.include heading-level=0}
introduction.md
getting-started.md
more-things.md
the-end.md
```

Note the heading-level=0 attribute. This tells MkTechDocs that the current heading level at the point of the include block is '0'. This will result in each included document starting with heading level of 1, or <H1> in HTML parlance. If you want to keep all your heading-level ones in the master document, that's okay, too. Simply use heading-level=1 when including subdocuments. More information on this follows.

2.3.2 Sub documents

Every sub document you include should start with a heading level of 1 (i.e. <H1>). MkTechDocs automatically increases the heading level of included documents (even multi-level recursive includes, where a document includes another document that includes another document, and so on). The heading-level adjustments will work provided that you give MkTechDocs a hint about the current heading level at the point of inclusion, as we have already seen:

### A level three header

```{.include heading-level=3}
myincludeddoc.md
```

Now, regardless of how many included documents appear in myincludedoc.md and in documents included in those documents, the heading levels should be consistent, provided you remember that all sub documents should start with a heading level of 1, as if they were a standalone document.

Tip: Creating every sub document such that it can exist on its own has the added benefit of allowing you to assemble different combinations of documents for different audiences. For example, you might want to build a small section of a much larger document as a standalone PDF.

2.3.2.1 Images

If you want to include images in your project document, include them inline using standard Markdown:

E.g.

```
Please click to [the following link](images/smiley.png) to see a smiley.
    
Or let's include it inline with a title underneath:
 
![Smiley](images/smiley.png)
```

And here's what the code above looks like after rendering:

Please refer to the following link to see a smiley.

Or let's include it inline with a title underneath:

Smiley

Then, tell MkTechDocs where your images directory is in the configuration section of your project makefile.

3 Building your document

3.1 From local install

To build your new project from scratch with the default configuration (a single CSS-styled HTML page with a navigation sidebar pinned to the left-hand side), simply do:

mktechdocs

If no errors occur, your new index.html HTML page and CSS file should appear in ./myproject/myproject_pages.

Here is the inline help:

$ mktechdocs help
Usage: mtechdocs [help|clean|init]
  help : Display this help message
  clean: Remove temporary build files
  init : Create a new MkTechDocs project in the current directory

Usage: mtechdocs
  Builds the MkTechDocs project in the current directory. Assumes
  that a mktechdocs.conf file and master.md file are present.

Usage: mtechdocs <config> <directory>
  Builds the MkTechDocs project in the given directory using the
  given configuration.
$

3.1.1 Multiple configurations

Note that a project can have as many configurations as necessary. For example, if you need to produce a PDF and styled webpage:

mktechdocs config1.conf /Users/mylogin/myproject
mktechdocs config2.conf /Users/mylogin/myproject

3.1.2 Pre- and post-build activities

Suppose you have a set of static documents as part of your documentation. You'll need to copy these to both the build and output directories in order for the images to appear inline.

Use the BUILD_SCRIPT configuration variable to provide the name of a script to run pre-build and post-build. The script should accept 2 arguments:

Here's an example build script:

#!/bin/bash

ACTIVITY=$1
OUT_DIR=$2

if [[ "$ACTIVITY" == "pre" ]] ; then
    echo "Creating some new stuff in ./mystuff..."
elif [[ "$ACTIVITY" == "post" ]] ; then
    echo "Copying ./mystuff to $OUT_DIR..."
    cp -r ./mystuff $OUT_DIR/.
fi

You don't need to worry about copying images to your output directory. Because images are often associated with documentation, MkTechDocs provides a configuration option where you can specify the path to your the directory that holds your images.

3.2 Docker

MkTechDocs is a complex set of code and dependencies, so it can be difficult to set up to run locally. Fortunately, MkTechDocs is available as a Docker image on Docker Hub: jsseidel/mktechdocs.

3.2.1 PDF output and docker

Please note that the Docker version of MkTechDocs does not currently support PDF output. Adding in this support makes the Docker image unreasonably large, and the image is already too big.

3.2.2 The build command

To get started, you'll need to have Docker installed. Then, building projects is relatively simple. When you first run the command below, MyMkTechDocsProject should be an empty directory. MkTechDocs will recognize that it should create a new project there. Subsequent runs will build your project.

docker run --user $(id -u):$(id -g) --rm -v /home/ubuntu/MyMkTechDocsProject:/project jsseidel/mktechdocs

Here's the breakdown of the command:

--user $(id -u):$(id -g): This tells docker to run the command under the current user and group id.

--rm: Remove the container after it exits. Otherwise, it will hang around in the Exited state, which can be useful if there were problems.

-v: Create a volume and map /home/ubuntu/MyMkTechDocsProject outside the container to /project inside the container. /project is where the entry script inside the container looks for a project to build.

jsseidel/mktechdocs: This is the docker image to use to run the container. Since no tag is specified (e.g. jsseidel/mktechdocs:0.0.1), docker will use the latest tag.

When you run it, an entry script inside the mktechdocs container will cd into /project and run mktechdocs. For this to work, your project must contain both a mktechdocs.conf and a master.md file.

3.2.3 Considerations

MkTechDocs uses a volume inside the Docker container (/project) and maps it to the contents to your MkTechDoc project directory. As such, it isn't possible to access directories outside your MkTechDoc project directory inside the container.

If you therefore have any pre- or post-build scripts, please note that the script will only have access to the contents of your MkTechDocs project directory.

For example, suppose you wanted to copy the contents of your output directory to some other directory and map that to your GitHub IO website. The following command will not work in a post-build script during a Docker build:

cp $OUTPUT_DIR/* ../docs/.

Inside the Docker container, when MkTechDocs is building, ../docs doesn't exist.

To fix this, you need docs inside your project directory. Then, the following will work:

cp $OUTPUT_DIR ./docs/.

4 Configuration

You configure MkTechDocs projects with variables contained in a configuration file in your project directory. By default, the name of this file is mktechdocs.conf, and if you choose to use this name, you won't have to provide the configuration file and directory when you run MkTechDocs.

4.1 Simple configuration

For the majority of relatively simple documents and websites, the simple configuration options should suffice.

4.2 Advanced configuration

More complex documents may require finer-grain control over how documents are rendered and stylized.

5 Title pages

There are three ways to create a title page for your pdf, pdffloat, docx, epub, and epub3 document. In all three cases, you must create a standalone file and assign that filename to the TITLE_PAGE variable in your project's configuration.

5.1 Title metadata block

% My Document Title
% John Doe; Jane Doe
% January 1, 2000

For more information about this syntax: man pandoc and search for "Metadata block."

5.2 YAML metadata block

---
title: My Document Title
author:
  - John Doe
  - Jane Doe
date: January 1, 2001
---

5.3 Templates

If your title page requires dynamic content (e.g. non-static date, version number) you can create a Python template to output information as in the above two examples. Follow along with the example in Python templates section.

6 Useful Blocks

MkTechDocs contains built-in blocks that are useful when creating documentation.

6.1 Comment

Comment blocks are ignored by MkTechDocs.

E.g.

```comment
This text will be ignored.
```

6.2 Include

The include block was introduced in the basics section.

E.g.

# My header

```{.include heading-level=1}
myinclude.md
mysecondinclude.md
```

include blocks support infinte recursion, so included documents can contain included documents and so on.

6.3 Include-code

The include-code block is used to include source-code files with syntax highlighting.

E.g.

```{.include-code language=c}
/home/mylogin/MkTechDocs/docs/file1.h
/home/mylogin/MkTechDocs/docs/file1.c
```

Produces:

#ifndef SOMETHING
typedef struct foo {
	int a;
} SOMETHING;
#endif

#include <stdio.h>
#include "file1.h"

int main(int argc, char **argv)
{
	printf("Size of SOMETHING is: %lu\n", sizeof(SOMETHING));
	return 0;
}

Note: Paths are an issue here. Because MkTechDocs projects are built inside your project directory, for files to be included correctly a full path or a path relative to the project directory must be given. MkTechDocs does its best to copy image files and directories to the output directory, but you may need to create a post-build script for more complex operations.

6.4 Note

note blocks are used to create documentation "notes" that appear highlighted in documentation text for HTML output formats. For PDF output, the content of the note block is prepended with "Note:".

E.g.

```note
This text will appear inside its own highlighted block and be prepended with "Note:".
```

Produces:

Note: This text will appear inside its own highlighted block and be prepended with "Note:".

6.5 Plantuml

plantuml blocks let you include PlantUML diagramming UML directly in your documentation.

E.g.

# My Diagram

```{.plantuml title="My Diagram"}
A->B    
```

Produces:

Note: TBD: This is currently not working.

6.6 Tip

tip blocks are used to create documentation "tips" that appear highlighted in documentation text for HTML output formats. For PDF output, the content of the tip block is prepended with "Tip:".

E.g.

```tip
This text will appear inside its own highlighted block and be prepended with "Tip:".
```

Produces:

Tip: This text will appear inside its own highlighted block and be prepended with "Tip:".

7 Document links

7.1 Pandoc links

Pandoc automatically creates anchor names for every heading within your documents. Therefore, generating internal links is theoretically easy thanks to Pandoc's naming convention. For example, suppose you create the following header:

# My New Header

Here's how you'd normally create a link to it using Pandoc naming conventions:

Some markdown text that contains a [link](#my-new-header)

7.2 MkTechDocs links

Because MkTechDocs can generate documents that require internal links that cross pages as well as documents that contain only internal relative links (as illustrated above), some thought has to be given to how links are created. When you generate internal links in your document, assume that your top-level included documents exist as HTML documents. For example, consider the following master document:

```{.include heading-level=0}
introduction.md
middle-section.md
end.md
```

To create links to other parts of your document, point your links to headings in introduction.html, middle-section.html, and end.html. For example, suppose introduction.md contained:

# Some heading

Some text

# Some other heading

Some more interesting text.

To refer to a section in this file in another file:

# Middle section

Here is a [link](introduction.html#some-heading) that points to another file.

If you follow this convention, MkTechDocs will produce correct links for every available format.

8 Python templates

MkTechDocs contains built-in support for Jinja2 templates. Using jinja2 templates is simply a matter of creating templates (plain text files containing Jinja2 markup code) and adding a .pyt extension, or, in the case of your header, landing page, and footer, a .htmlt extension, so MkTechDocs knows how to build them.

To use templates in MkTechDocs, you'll need to create both a renderer and a template to render.

8.1 The template

A template is nothing more than plain text with some special Jinja2 markup thrown in. For example, let's create a dynamic title page using templates. First, we'll create a template that represents a dynamic YAML title block in our project directory:

title-page.pyt:

---
title: My Title {{ docVersion }}
author:
    - John Doe
    - Jane Doe
date: Built on {{ currentDateTime }}
---

The Jinja2 templating engine will replace the variables in the double curly brackets with values you provide in the renderer.

8.2 The renderer

Now, in your project directory, create a renderer with the same name as the template, but replace .pyt with .renderer. MkTechDocs will automatically use the renderer associated with the template name. Renderers are built with pure Python code. Continuing with our title-page example, here is a renderer that will properly render our title page template.

title-page.renderer:

#!/usr/bin/env python

import os
import sys
import datetime
from mktechdocslib import render_mktechdocs_jinja2_template

##
# Create a simple renderer function that ouputs a template passed in on the
# command line with the given variable dictionary.
#

def render():
  # Here, we hard code a document version, but in a real-world example, we'd
  # probably want to grab this from a configuration file or even a database.
  docVersion = "0.1a"

  # Now set up the date and time however you see fit
  currentDateTime = datetime.datetime.now().strftime("%I:%M%p on %B %d, %Y")

  varDictionary = {"docVersion":docVersion, "currentDateTime":currentDateTime}

  if not os.path.isfile(sys.argv[1]):
    sys.stderr.write("Cannot find " + sys.argv[1] + "\n")
    sys.exit(1)

  print render_mktechdocs_jinja2_template(sys.argv[1], varDictionary)

if __name__ == "__main__":
  render()

You can add as many variables to your varDictionary as necessary. You can even add entire modules and functions, if you need more logic in your templates.

Next, set the TITLE_PAGE variable in your project makefile to title-page.md. Why the .md extension? MkTeckDocs automatically converts anything with a .pyt extension to markdown before building your documents. Here's a diagram that illustrates the process:

Since title pages are only used in PDF, docx, and epub documents, you should also set FORMAT to pdf, pdffloat, docx, epub, or epub3.

8.3 Referencing templates in include blocks

MkTechDocs automatically detects *.pyt templates and converts them into markdown before processing include blocks. So, if you need to include the contents of a template's output in another markdown file, simply treat the template as if it were markown. For example, if your template's name is mytemplate.pyt reference the output in include blocks like:

```include
mytemplate.md
```

8.4 footer.html, header.html, and landing.html as templates

MkTechDocs will recognize footer.htmlt, header.htmlt, and landing.htmlt as templates and process into corresponding .html files, using similarly named renderers. E.g. footer.renderer.

8.5 Escaping curly brackets

Because every MkTechDocs document is ultimately converted into a large Python Jinja2 template, if you want to include double curly brackets as curly brackets in your document, for example, if you're documenting Jinja2, you will need to escape them.

Here's how:

    {% raw %}
 
    {{somevar}}
 
    {% endraw %}

If you need to escape a single curly bracket, you can use:

{{openCurlyBracket}} and {{closeCurlyBracket}}

8.6 For more information

For real-world examples of how to use Jinja2 templates with MkTechDocs, see the following files in the MkTechDocs directory:

docsbuild/runningfilters.pyt
docsbuild/runningfilters.renderer
docsbuild/scripts.pyt
docsbuild/scripts.renderer

These templates and renderers incorporate some pythondoc-like markup in the filters and scripts that come with MkTechDocs.

Jinja2 templates can be powerful documentation tools. Please reference the Jinja2 template documentation for more information 2>$template.err.

9 Managing document versions

For some documentation projects, managing document versions is vital. It is helpful if you define your document version in one place, so that any sub-documents that reference the version number get it from the same place.

The following is one reasonable way of doing this.

9.1 global_vars

Whether you're using Python or Groovy, it's probably a good idea to create a module (file) called global_vars.py to hold any static variables or functions you need to build your project, such as a version string. Here are the steps:

1. Add your project's directory to PYTHONPATH.
2. For any documents that need access to information stored in global_vars, you simply convert the document into a template (by renaming it from .md to .pyt, for example).
3. import global_vars as a module in your renderer and include it in your variable dictionary.
4. Now, you can access global variables and functions in glob.py in your templates that have imported it: e.g. version=global_vars.VERS.

10 Incorporating Groovy code and templates

Groovy integration is an optional step and only necessary if you want to access existing Java libraries.

10.1 Installation of groovy components

First, start by examining and then running whichever install_deps_*_groovy.sh script in $MKTECHDOCSHOME/bin/groovy is appropriate for your architecture.

They install two packages, groovy and gradle. Then, they download a Groovy library called groovy-pandoc from GitHub and install the resulting jar file in $MKTECHDOCSHOME/lib.

10.1.1 Environment

Next, add $MKTECHDOCSHOME/bin/groovy to your PATH environment variable. Here's an example in Bash:

export PATH=$PATH:$MKTECHDOCSHOME/bin/groovy

10.1.2 Testing the integration

Finally, run $MKTECHDOCSHOME/test/testgroovyintegration.sh. Your output should look something like this:

[~/MkTechDocs/test]$ testgroovyintegration.sh
Generating 'svg' plantuml diagram named 'Diagram'
Test dependency installation
============================

First, we'll try an include:

Include test file
-----------------

This text is coming from inside test\_install\_include.md.

Now, we'll try some plantUML:

![Diagram](./8410b9ad92ce7f3e2d564faff2b834b1.plantuml.svg){#myid}
Pandoc seems ok.


Checking groovy...

If this prints with no error, you are good to go
Groovy seems ok.
[~/MkTechDocs/test]$

10.2 Groovy templates

You can use Groovy templates (*.gt) as you would Jinja2 Python templates, except that Groovy templates do not require a renderer, which makes them somewhat more convenient. However, this comes at a price. Groovy templates are far slower than Jinja2 to generate, even with GroovyServ installed.

Groovy templates work like PHP or JSP files. You can include plain text markdown and Groovy clode blocks. Here's an example:

# An example Groovy template

Let's count to 10:

<%
(1..10).each { n ->
    out.println(n)
}
%>

Here's the output:

$ gtp testtemplate.gt
# An example Groovy template

Let's count to 10:

1
2
3
4
5
6
7
8
9
10


$

Notice that gtp is a standalone template renderer that MkTechDocs uses to render *.gt files.

10.3 Groovy classes

You might need standalone classes for your Groovy templates. To do this, create *.groovy files in your documentation directory. MkTechDocs will automatically detect and compile these into *.class files.

For example, MkTechDocs includes a class called MarkdownUtils, a utility class. Even though you wouldn't likely use it in a real project, the following demonstrates the concept.

test.md:

# This is a level-one header

## This is a level-two header

mytempl.gt:

# Some heading

<%
import MarkdownUtils

def f = new File("./test.md")

out << MarkdownUtils.getMarkdownFileAndChangeHeader(f, 2)
%>

And here is the output:

[~/MkTechDocs/docs]$ gtp mytempl.gt
# Some heading

### This is a level-one header

#### This is a level-two header


[~/MkTechDocs/docs]$

Note how the heading levels were increased by two levels, which is exactly what the getMarkdownFileAndChangeHeader function does.

10.4 CLASSPATH

MkTechDocs creates a CLASSPATH automatically in order to process templates. However, if you need to add jars or directories to your CLASSPATH outside of MkTechDocs, simply create an environment variable. Here is an example in Bash:

export CLASSPATH=/path/to/my.jar

Now, when MkTechDocs runs, it will build a CLASSPATH that includes any existing CLASSPATH.

11 Team documentation

MkTechDocs is ideal for incorporating team-contributed documentation. Because MkTechDocs documentation is simple Markdown, team members can contribute or edit documents via Git (for example) and let a single technical writer manage the pull requests to keep things as simple and as clear as possible.

This makes scaling documentation easier because the technical writer is no longer limited to his or her own domain knowledge or learned knowledge.

That's a simple case. The following describes a different scenario that lets agile developers contribute to documentation in a more specific and controlled way.

11.1 Case study

In this study, a number of different agile development teams are responsible for individual components of a larger system. A configuration repository maintained by the larger system consists of a number of YAML files that describe the general deployment of each component.

Here, components could be Docker containers, VMs, micro-services, or whatever. Each component could have a different but similar set of deployment instructions for each category below, or it could simply conform to some generic set of instructions and require no extra documentation.

1. Overview
2. Environmental-setup
3. Pre-deployment
4. Deployment
5. Undeployment
6. Upgrading
7. Jenkins integration

Agile team members in this scenario are encouraged to contribute documentation in support of their respective components with those steps in mind. If they feel that specialized instructions for any of the above areas applies to their component, they should create a file in a provided directory within the documention tree that matches their component name in the configuration YAML. This file should contain a standalone Markdown document containing instructions specific for their component for that particular step. E.g. overview/mycomponent.md, upgrading/mycomponent.md.

For example, suppose a VM named vm-inventory exists in some YAML configuration file:

location:
  description: >
    This is a sample description of a particular location
  vm-deployments:
         .
         .
         .
    vm-inventory:
      vm-config: vmi.cfg
      release: 1.234
      notes: Some notes about this particular VM
        .
        .
        .

Now, we can create a template that loads the YAML containing the list of components, and looks for files with the same name in the directories listed above. If we find specialized instructions, we output them for a particular stage (e.g. Overview, Undeployment). If we find no specialized instructions, we output some generic ones.

11.2 Managing git commit messages

Often, it is useful for your document's audience to understand what has changed from version to version. Rather than keeping track of this manually, in a "change log," for example, you can leverage Git's commit system.

The git log command shows a list of all commits to a respository. Each commit has a unique ID associated with it (SHA1 collisions not withstanding). This means that you can create "versions" by recording specific commit messages that represent the beginning and ending of a document version, pulling out all relevant commit messages dynamically when the document is built.

MkTechDocs includes this functionality in the mktechdocslib.py module. Here's an example of how to use it.

11.2.1 GitCommitDB

What we'd like to do is create a simple database of Git messages that exist between two arbitrary Git commit IDs. First, we examine the output of git log to determine the first and last commit IDs that frame the version of the document we're interested in creating. Note that the output of git log is ordered by commit date, so the most recent dates appear first.

Here is a sample Git log:

commit 20de5c8bf2a53efe54a71ae06e5b0b2a813d5176
Author: Seidel, Joseph (js2589) <spence@research.att.com>
Date:   Tue Mar 7 08:04:25 2017 -0500

    Fixed several formatting issues

commit 3523b7f36d928519bba9568852babd793e783a7c
Merge: cd6c4bc f9b3edf
Author: Seidel, Joseph (js2589) <spence@research.att.com>
Date:   Tue Mar 7 06:50:53 2017 -0600

    Merge pull request #71 in FOO_BAR/foo.bar.documentation from feature/FOOBAR/FOOBAR to master

    * commit 'f9b3edfb5194a2be0456d26d742043677948d5ea':
      Added some notes about documentation formats

commit f9b3edfb5194a2be0456d26d742043677948d5ea
Author: Seidel, Joseph (js2589) <spence@research.att.com>
Date:   Mon Mar 6 16:21:17 2017 -0500

    Added some notes about documentation formats

From this, we determine that the first commit ID is f9b3edfb5194a2be0456d26d742043677948d5ea and the last commit ID is 20de5c8bf2a53efe54a71ae06e5b0b2a813d5176.

Now, in a template.renderer, we can do something like this:

#!/usr/bin/env python

import os
import sys
import datetime
import global_vars
from mktechdocslib import GitCommitDB, render_mktechdocs_jinja2_template

##
# Create a simple renderer function that ouputs a template passed in on the
# command line with the given variable dictionary.
#

def render():
    messages = GitCommitDB("f9b3edfb5194a2be0456d26d742043677948d5ea", "20de5c8bf2a53efe54a71ae06e5b0b2a813d5176").messages
    varDictionary = {"gitMessages":messages,
                     "numCommits":len(messages)}
    
    if not os.path.isfile(sys.argv[1]):
        sys.stderr.write("Cannot find " + sys.argv[1] + "\n")
        sys.exit(1)

    print render_mktechdocs_jinja2_template(sys.argv[1], varDictionary)

if __name__ == "__main__":
    render()

Here, we import GitCommitDB from mktechdocslib and add the GitCommitDB.messages list object to our variable dictionary. messages contains a list of GitCommitMessage objects, which have the following properties:

Finally, here's our template that displays the information we're interested in:

# Document change log

{% if numCommits == 0 %}
Looks like no changes have been committed.
{% endif %}

{% for m in gitMessages %}

{% if not m.mergeCommit %}
**Date**: {{ m.date }}<br />
**Author**: {{ m.author }}<br />
**Message**: {{ m.message }}<br />
{% endif %}

{% endfor %}

The version in the example above was hard coded. In reality, it would be better to keep track of version information in a list of tuples. For example:

versions.py:

gVersions = [("1.0", "2017-04-06", "f9b3edfb5194a2be0456d26d742043677948d5ea", "20de5c8bf2a53efe54a71ae06e5b0b2a813d5176"),
             ("0.9", "2017-03-25", "d453519c024a49806fbd3a6740a9f0f586aaafbc", "57831131f186a60b864af27e53e06d2772a6d1ef")]

In this scheme, the latest version tuple would always be gVersions[0]. Or, if you prefer to store the versions in reverse order: gVersions[len(gVersions)-1]. Then, our renderer would look like this:

#!/usr/bin/env python

import os
import sys
import datetime
import global_vars
from versions import gVersions
from mktechdocslib import GitCommitDB, render_mktechdocs_jinja2_template

##
# Create a simple renderer function that ouputs a template passed in on the
# command line with the given variable dictionary.
#

def render():
    (versString, date, fcid, lcid) = gVersions[0]
    
    messages = GitCommitDB(fcid, lcid).messages
    
    varDictionary = {"gitMessages":messages,
                     "numCommits":len(messages),
                     "versString":versString,
                     "date":date}
    
    if not os.path.isfile(sys.argv[1]):
        sys.stderr.write("Cannot find " + sys.argv[1] + "\n")
        sys.exit(1)

    print render_mktechdocs_jinja2_template(sys.argv[1], varDictionary)

if __name__ == "__main__":
    render()

12 Install MkTechDocs

12.1 Docker

The easiest way to use MkTechDocs is via Docker.

docker run --rm --user $(id -u):$(id -g) -v $PROJECTDIR:/project jsseidel/mktechdocs

If $PROJECTDIR is an empty directory, MkTechDocs will create a new project. If $PROJECTDIR already contains a MkTechDocs project, MkTechDocs will build it.

Note: The Docker version of MkTechDocs does not support PDFs.

12.2 PPA

Add and install the Launchpad PPA in Ubuntu 18.04+:

sudo add-apt-repository -y ppa:jsseidel/mktechdocs
sudo apt update
sudo apt install -y mktechdocs

Then:

. /opt/mktechdocs/bin/mktechdocs.env

12.3 Deb package

Download the latest deb package from the MkTechDocs GitHub releases page: https://github.com/att/MkTechDocs/releases.

sudo apt install -y ./mktechdocs_18.04_1.0.8_amd64.deb

Then:

. /opt/mktechdocs/bin/mktechdocs.env

12.4 Source

To download the MkTechDocs source tree, clone the repo:

git clone https://github.com/att/MkTechDocs

Then, set up your environment.

13 Setting up your environment

Please export the following variables. Here is an example in Bash:

# MkTechDocs libraries and scripts require the following variable:
export MKTECHDOCSHOME=/path/to/cloned/repo/MkTechDocs

# Make sure the MkTechDocs bin directory is in your path. Here we add the current directory as well, `.`, because we want MkTechDocs to pick up templates in the project directories:
export PATH=$PATH:$MKTECHDOCSHOME/bin:.

# Let Python know where to find MkTechDocs libraries:
export PYTHONPATH=$MKTECHDOCSHOME/bin

If you installed MkTechDocs from a Debian package or Launchpad PPA, you can simply source the following to add these variable to your shell environment:

. /opt/mktechdocs/bin/mktechdocs.env

13.1 Running the dependencies installation script

Install MkTechDocs dependencies using the (admittedly imperfect) installation script. Consider the script more of a guide.

Examine the contents of the $MKTECHDOCSHOME/bin/install_deps*sh script appropriate for your architecture (Ubuntu, Arch, Fedora, and macOS are currently supported). For Groovy support, please also see the install_deps*sh scripts in the bin/groovy directory. The install script simply tries to install the various dependencies using an architecture-appropriate package manager and in some cases checks version numbers of installed binaries. You can also install the dependencies manually.

13.1.1 Dependencies

MkTechDocs requires the following:

** homebew: macOS only

Groovy/Java*

*: Java/Groovy dependencies are optional but ideal for interfacing MkTechDocs with existing Java libraries. Note that Groovy pandoc filters are orders of magnitude slower than their Python counterparts. To shorten build time, you might want to try GroovyServ.

13.2 Testing your environment

After installing the dependencies, run this:

cd $MKTECHDOCSHOME/test
./testinstall.sh

You should see something like this:

[~/MkTechDocs/test]$ ./testinstall.sh
Created image plantuml-c6117aebb4ce8e59ba2b46950eca869f.svg
Test dependency installation
============================

First, we'll try an include:

Include test file
-----------------

This text is coming from inside test\_install\_include.md.

Now, we'll try some plantUML:

![](plantuml-c6117aebb4ce8e59ba2b46950eca869f.svg)
Pandoc seems ok.


Checking python...

If this prints with no error, you are good to go!
Python seems ok.
[~/MkTechDocs/test]$

If you get errors, examine the output of the installation script to see where something went wrong.

13.3 A note about titlesec

MkTechDocs uses a TeX package named titlesec to do various things with PDF section numbers (e.g. 6.1.3.2). Unfortunately, Ubuntu 16.04 LTS ships with a buggy version of this package, which causes section numbers to be empty in various situations, so replacing it will be necessary if you plan to use section numbering in PDF documents. To replace it, first download the latest titlesec from https://www.ctan.org/tex-archive/macros/latex/contrib/titlesec.

Now:

unzip titlesec.zip
sudo rm -rf /usr/share/texlive/texmf-dist/tex/latex/titlesec
sudo mkdir /usr/share/texlive/texmf-dist/tex/latex/titlesec
sudo cp -r titlesec/* /usr/share/texlive/texmf-dist/tex/latex/titlesec/.

Section numbering should work as expected.

14 Running filters

Running MkTechDocs Pandoc filters outside of the normal build environment is easy if you've set up your environment properly:

pandoc --filter <flt-filtername.py> -f markdown -t some-other-format inputfile.md >outputfile.fmt

In some cases, MkTechDocs filters produce artifacts on STDERR. In these cases use something like the following:

pandoc --filter <flt-filtername.py> -f markdown -t some-other-format inputfile.md >outputfile.fmt 2>artifacts.out

The following Pandoc python filters come with MkTechDocs. Groovy versions are available as well in $MKTECHDOCSHOME/bin/groovy. They are run in the same way (but have a .groovy extension).

14.1 flt-comment-block.py

Adapted from https://github.com/jgm/pandocfilters/blob/master/examples/comments.py

Pandoc filter that causes everything between '<!-- BEGIN COMMENT -->' and '<!-- END COMMENT -->' to be ignored. The comment lines must appear on lines by themselves, with blank lines surrounding them.

E.g.

 <!-- BEGIN COMMENT -->

 This text will be ignored.

 <!-- END COMMENT -->

14.2 flt-comment.py

Ignores all text contained in the comment block. E.g.

 ```comment
 This text will be ignored.
 ````

14.3 flt-decrement-header-1.py

Decrements all headers by 1 level

14.4 flt-decrement-header-2.py

Decrements all headers by 2 levels

14.5 flt-decrement-header-3.py

Decrements all headers by 3 levels

14.6 flt-decrement-header-4.py

Decrements all headers by 4 levels

14.7 flt-decrement-header-5.py

Decrements all headers by 5 levels

14.8 flt-get-includes.py

Outputs all files referenced in any include blocks in a file to stderr. Used internally by MkTechDocs to build dependency lists.

14.9 flt-include-code.py

This filter allows you to include source-code files in your markdown.

You can provide one file per line. If you include the "language" property:

 ```{.include-code language="java"}
 myfile.java
 myfile.sh
 ```

The filter will use whatever language you specify in the property regardless of what the file extensions are, which you may or may not want.

If you provide something like the following:

 ```include-code
 myfile.java
 myfile.bash
 myfile.c
 ```

The filter will produce 3 code blocks with the language property automatically set based on the extension of the files within.

14.10 flt-include-doc-map.py

This filter is identical to the include filter, except that while building a document, it outputs a document map on stderr so that a script can figure out where each part of the document came from. E.g.

     ```include
     includethisfile.md
     ```

This filter is recursive, so you markdown can include other markdown to any level.

This filter is used internally by MkTechDocs.

14.11 flt-include.py

This filter allows you to include other markdown in your markdown. E.g.

     ```include
     includethisfile.md
     ```

This filter is recursive, so your markdown can include other markdown to any level.

Header levels are automatically adjusted to the correct heading level if you use the "heading-level" attribute:

 # Heading One

 Some text

 ## Heading Two

 ```{.include heading-level=2}
 somefiletoinclude.md
 ````

"heading-level" represents the heading level where the include happens. On subsequent inclusion, the heading levels in the included file will be increased by two. Of course, you can always leave out the heading-level if you don't want to alter the heading level of the included file.

Even multiple levels of include recursion will result in headers of the correct level. The reason this works is by virtue of that depth-first recursion.

The lowest level pages (i.e. the last file included in a recursive string of includes), are processed and stored to disk first.

When the next outer level is processed, those headers are adjusted and the previously adjusted levels in the included file are in effect adjusted again. Then, the next level is adjusted, and so are all the inclusions.

This results in the innermost include's headers being adjusted upwards by however many levels of recursion there are.

14.12 flt-increment-header-1.py

Increments all headers by 1 level to a maximum of 6 levels.

14.13 flt-increment-header-2.py

Increments all headers by 2 levels to a maximum of 6 levels.

14.14 flt-increment-header-3.py

Increments all headers by 3 levels to a maximum of 6 levels.

14.15 flt-increment-header-4.py

Increments all headers by 4 levels to a maximum of 6 levels.

14.16 flt-increment-header-5.py

Increments all headers by 5 levels to a maximum of 6 levels.

14.17 flt-notetip.py

This filter replaces 'note' and 'tip' blocks with div tags of either a note or tip class

E.g.

 ```tip
 My tip
 ```

Becomes:

 <div class='tip'>**Tip**: My tip</div>

for markdown and html. For other output formats:

TIP: My tip

14.18 flt-plantuml.py

Adapted from: https://github.com/jgm/pandocfilters/blob/master/examples/plantuml.py

Turns PlantUML, e.g.:

 ```plantuml
 actor Foo1
 boundary Foo2
 control Foo3
 entity Foo4
 database Foo5
 Foo1 -> Foo2 : To boundary
 Foo1 -> Foo3 : To control
 Foo1 -> Foo4 : To entity
 Foo1 -> Foo5 : To database
 ````

Into an image for inclusion in HTML documents.

To affect the output format, provide a metadata variable on the command line:

 pandoc -M umlformat=[eps,svg] . . .

If no umlformat is indicated, the filter will default to svg.

Needs plantuml.jar from http://plantuml.com/.

14.19 flt-strip-pages-from-links.py

This filter turns links like this:

 [foo link](somepage.html#some-heading)

into:

 [foo link](#some-heading)

This is necessary to transform documents from multi-page to single page

14.20 flt-template.py

A starting point for building new filters.

15 Scripts

Several potentially useful scripts come with MkTechDocs.

15.1 dec-headers.sh

A utility script that decrements the headers in a given markdown file by a given number of levels.

A copy of the original file is kept using the origial file name and a PID as the extension.

15.2 escape-jinja-brackets.sh

This script escapes opening Jinja2 template brackets, {# to avoid internal build problems. Generally, you won't need to run this script directly.

15.3 inc-headers.sh

A utility script that increments the headers in a given markdown file by a given number of levels.

A copy of the original file is kept using the origial file name and a PID as the extension.

15.4 install_deps_1804+_py.sh

None

16 Addendum

16.1 Understanding the build process

The MkTechDocs build process is complex. The following is a step-by-step guide towards understanding how it works.

1. MkTechDocs consists of a BASH control script and a library of Python and Groovy filters.
2. The control script is responsible for importing the project configuration and calling pandoc with the correct arguments to build reasonably formatted documents in a variety of formats.
3. The "magic" behind it all is pandoc 's Abstract Syntax Tree (AST). Internally, pandoc converts all documents into AST, which can then be converted to JSON, which can then be manipulated by filters.
4. MkTechDocs exploits this with a couple of libraries, GroovyPandoc, for Groovy template integration, and PandocFilters, for Python integration.
5. Every time you run mktechdocs, the control script calls pandoc on your project's master document. Some Bash scripting then takes care of some little details, which you can see for yourself by examining the script.

16.2 Contact

MkTechDocs was built and is maintained by Spencer Seidel.

17 Document change log

Date: Tue Feb 18 09:30:41 2020 -0500
Author: jsseidel jsseidel@fastmail.com
Message: Put docker builds and normal builds under same heading. [doc]

Date: Sat Apr 20 16:50:43 2019 -0400
Author: jsseidel jsseidel@fastmail.com
Message: Updated supported architectures section and reformatted a few paragraphs. [doc]

Date: Sat Apr 20 14:21:31 2019 -0400
Author: jsseidel jsseidel@fastmail.com
Message: Added section about how to install MkTechDocs from PPA, deb, and Docker. [doc]

Date: Fri Apr 19 15:14:08 2019 -0400
Author: jsseidel jsseidel@fastmail.com
Message: Updated to include better information about docker builds [doc]

Date: Fri Jan 25 16:35:25 2019 -0500
Author: jsseidel jsseidel@fastmail.com
Message: Added bit about putting the current directory in the path when setting up the environment to build. [doc]

Date: Thu Jan 17 11:12:22 2019 -0500
Author: jsseidel jsseidel@fastmail.com
Message: Fixed missing import in templates sample code [doc]

Date: Sat Sep 15 14:11:33 2018 -0400
Author: jsseidel jsseidel@fastmail.com
Message: Added a caveat about docker builds [doc]

Date: Sat Sep 15 13:53:53 2018 -0400
Author: jsseidel jsseidel@fastmail.com
Message: Updated build script docs. It was out of date. [doc]

Date: Fri Jul 6 10:31:15 2018 -0400
Author: jsseidel jsseidel@fastmail.com
Message: Added section about how to escape curly brackets [doc]

Date: Thu Jul 5 15:03:43 2018 -0400
Author: jsseidel jsseidel@fastmail.com
Message: Updated to point to tutorial MkTechDocs project [doc]

Date: Thu Jul 5 11:15:26 2018 -0400
Author: jsseidel jsseidel@fastmail.com
Message: Added section for doing docker builds [doc]

Date: Mon May 21 12:26:48 2018 -0400
Author: jsseidel jsseidel@fastmail.com
Message: Added a 'doesn't work' note to the section about plantuml images titles. [doc]

Date: Mon May 21 11:41:47 2018 -0400
Author: jsseidel jsseidel@fastmail.com
Message: Added bit about 18.04 and cleaned up some wording [doc]

Date: Tue Apr 24 15:58:05 2018 -0400
Author: jsseidel jsseidel@fastmail.com
Message: Updated version to 1.0 and reset document change log [doc]

Date: Tue Apr 24 15:13:25 2018 -0400
Author: jsseidel jsseidel@fastmail.com
Message: Changed build system to remove makefile and updated documentation [doc]