1 Introduction

1.1 What is it?

MkTechDocs is an open framework for collating and transforming markdown. You can use it to manage complex technical documentation projects with dependencies, produce light documentation, or combine and transform existing Markdown files into a styled webpage or PDF document.

1.2 Features

1.3 Can I see an example?

You’re looking at one!

1.4 Supported architectures

MkTechDocs was built in and is most frequently used in Ubuntu Linux 18.04 LTS; however, MkTechDocs should work in any environment that supports these dependencies.

1.5 Install

See the Install MkTechDocs and Setting up your environment sections at the end of this document for more information.

2 The basics

The easiest way to learn how to use MkTechDocs effectively is with a tutorial project. Here’s one on GitHub.

Unlike the comprehensive usage guide, the tutorial project uses a Docker image to build and provides a good overview of a MkTechDocs project without getting too far into the weeds.

2.1 Markdown

MkTechDocs uses Markdown exclusively, although support for additional syntaxes may be added in the future. More specifically, it uses standard Markdown syntax with Pandoc extensions. In this documentation, assume that any file with a .md extension is a file that contains Markdown. Markdown is easy to learn, read, and edit and is easily transformable into a myriad of output formats. To use MkTechDocs effectively, take a few minutes to familiarize yourself with Markdown syntax.

2.2 Creating a new project

To create a new MkTechDocs project, cd inside the directory where you want to create it and run mktechdocs. For example:

mkdir mynewproject
cd mynewproject
mktechdocs init

mktechdocs will create the following files in mynewproject:

File Name Purpose
footer.html* Custom HTML content MkTechDocs should place in the footer area of HTML output.
header.html* Custom HTML content MkTechDocs should place in the header area of HTML output.
landing.html* Custom HTML landing-page content MkTechDocs uses in index.html when creating multi-html-page output.
master.md A stub master document.
subdocument.md An example subdocument that’s included inside the master.
mktechdocs.conf Project configuration.

* NOTE 1: Optional. If you don’t need this HTML content, for example, if you’re producing PDF content only, you can safely delete these files.

* NOTE 2: If you need more dynamic content in your header, footer, or main content, dates, database information, etc., please read the footer.html, header.html, and landing.html as templates section

2.3 MkTechDocs projects

2.3.1 Master document

MkTechDocs projects consist of a “master” document — a single markdown document that defines your project, always named master.md— and any number of sub documents. Building a master document is simple:

```comment
Define one or more include blocks to include all the major subdocuments that
make up your project.
```
```{.include heading-level=0}
introduction.md
getting-started.md
more-things.md
the-end.md
```

Note the heading-level=0 attribute. This tells MkTechDocs that the current heading level at the point of the include block is ‘0’. This will result in each included document starting with heading level of 1, or <H1> in HTML parlance. If you want to keep all your heading-level ones in the master document, that’s okay, too. Simply use heading-level=1 when including subdocuments. More information on this follows.

2.3.2 Sub documents

Every sub document you include should start with a heading level of 1 (i.e. <H1>). MkTechDocs automatically increases the heading level of included documents (even multi-level recursive includes, where a document includes another document that includes another document, and so on). The heading-level adjustments will work provided that you give MkTechDocs a hint about the current heading level at the point of inclusion, as we have already seen:

### A level three header

```{.include heading-level=3}
myincludeddoc.md
```

Now, regardless of how many included documents appear in myincludedoc.md and in documents included in those documents, the heading levels should be consistent, provided you remember that all sub documents should start with a heading level of 1, as if they were a standalone document.

Tip: Creating every sub document such that it can exist on its own has the added benefit of allowing you to assemble different combinations of documents for different audiences. For example, you might want to build a small section of a much larger document as a standalone PDF.

2.3.2.1 Images

If you want to include images in your project document, include them inline using standard Markdown:

E.g.

```
Please click to [the following link](images/smiley.png) to see a smiley.
    
Or let's include it inline with a title underneath:
 
![Smiley](images/smiley.png)
```

And here’s what the code above looks like after rendering:


Please refer to the following link to see a smiley.

Or let’s include it inline with a title underneath:

Smiley

Then, tell MkTechDocs where your images directory is in the configuration section of your project makefile.

3 Building your document

3.1 From local install

To build your new project from scratch with the default configuration (a single CSS-styled HTML page with a navigation sidebar pinned to the left-hand side), simply do:

mktechdocs

If no errors occur, your new index.html HTML page and CSS file should appear in ./myproject/myproject_pages.

Here is the inline help:

$ mktechdocs help
Usage: mtechdocs [help|clean|init]
  help : Display this help message
  clean: Remove temporary build files
  init : Create a new MkTechDocs project in the current directory

Usage: mtechdocs
  Builds the MkTechDocs project in the current directory. Assumes
  that a mktechdocs.conf file and master.md file are present.

Usage: mtechdocs <config> <directory>
  Builds the MkTechDocs project in the given directory using the
  given configuration.
$

3.1.1 Multiple configurations

Note that a project can have as many configurations as necessary. For example, if you need to produce a PDF and styled webpage:

mktechdocs config1.conf /Users/mylogin/myproject
mktechdocs config2.conf /Users/mylogin/myproject

3.1.2 Pre- and post-build activities

Suppose you have a set of static documents as part of your documentation. You’ll need to copy these to both the build and output directories in order for the images to appear inline.

Use the BUILD_SCRIPT configuration variable to provide the name of a script to run pre-build and post-build. The script should accept 2 arguments:

Argument # Description
1 The activity being performed. Possible values given by MkTechDocs are pre and post. These indicate that any pre- or post-build activies should take place.
2 The full path to the output directory.

Here’s an example build script:

#!/bin/bash

ACTIVITY=$1
OUT_DIR=$2

if [[ "$ACTIVITY" == "pre" ]] ; then
    echo "Creating some new stuff in ./mystuff..."
elif [[ "$ACTIVITY" == "post" ]] ; then
    echo "Copying ./mystuff to $OUT_DIR..."
    cp -r ./mystuff $OUT_DIR/.
fi

You don’t need to worry about copying images to your output directory. Because images are often associated with documentation, MkTechDocs provides a configuration option where you can specify the path to your the directory that holds your images.

3.2 Docker

MkTechDocs is a complex set of code and dependencies, so it can be difficult to set up to run locally. Fortunately, MkTechDocs is available as a Docker image on Docker Hub: jsseidel/mktechdocs.

3.2.1 PDF output and docker

Please note that the Docker version of MkTechDocs does not currently support PDF output. Adding in this support makes the Docker image unreasonably large, and the image is already too big.

3.2.2 The build command

To get started, you’ll need to have Docker installed. Then, building projects is relatively simple. When you first run the command below, MyMkTechDocsProject should be an empty directory. MkTechDocs will recognize that it should create a new project there. Subsequent runs will build your project.

docker run --user $(id -u):$(id -g) --rm -v /home/ubuntu/MyMkTechDocsProject:/project jsseidel/mktechdocs

Here’s the breakdown of the command:

--user $(id -u):$(id -g): This tells docker to run the command under the current user and group id.

--rm: Remove the container after it exits. Otherwise, it will hang around in the Exited state, which can be useful if there were problems.

-v: Create a volume and map /home/ubuntu/MyMkTechDocsProject outside the container to /project inside the container. /project is where the entry script inside the container looks for a project to build.

jsseidel/mktechdocs: This is the docker image to use to run the container. Since no tag is specified (e.g. jsseidel/mktechdocs:0.0.1), docker will use the latest tag.

When you run it, an entry script inside the mktechdocs container will cd into /project and run mktechdocs. For this to work, your project must contain both a mktechdocs.conf and a master.md file.

3.2.3 Considerations

MkTechDocs uses a volume inside the Docker container (/project) and maps it to the contents to your MkTechDoc project directory. As such, it isn’t possible to access directories outside your MkTechDoc project directory inside the container.

If you therefore have any pre- or post-build scripts, please note that the script will only have access to the contents of your MkTechDocs project directory.

For example, suppose you wanted to copy the contents of your output directory to some other directory and map that to your GitHub IO website. The following command will not work in a post-build script during a Docker build:

cp $OUTPUT_DIR/* ../docs/.

Inside the Docker container, when MkTechDocs is building, ../docs doesn’t exist.

To fix this, you need docs inside your project directory. Then, the following will work:

cp $OUTPUT_DIR ./docs/.

4 Configuration

You configure MkTechDocs projects with variables contained in a configuration file in your project directory. By default, the name of this file is mktechdocs.conf, and if you choose to use this name, you won’t have to provide the configuration file and directory when you run MkTechDocs.

4.1 Simple configuration

For the majority of relatively simple documents and websites, the simple configuration options should suffice.

Variable Name Possible Values Effects
TITLE Any The natural-language title of your project.
OUTPUT_FILE_NAME Any. No spaces or punctuation. Certain outputs will produce a file of this name.
FORMAT pdf, pdffloat, html, htmlsimple, cssframes, htmlmulti, markdown, docx, epub, epub3 Output format

pdf: A PDF file with section numbers and a title page. Diagrams are positioned near where they are included.
pdffloat: A PDF file with section numbers and a title page. Diagrams are “floated” within the document.
html: A single CSS-styled HTML page with a hideable table of contents at the top.
htmlsimple: A single unstyled HTML page with no table of contents.
cssframes: A single CSS-styled HTML page with a CSS-locked navigation sidebar on the left-hand side and locked content on the right-hand side. A customizable header and footer (in header.html and footer.html) are used to frame the page.
htmlmulti: Multi-page CSS-styled HTML with a CSS-locked navigation bar on the left-hand side and content on the right-hand side. A customizable header and footer (in header.html and footer.html) are used to frame the page. In addition, landing.html is used as a landing page with the CSS-locked navigation sidebar on the left-hand side. Best for large projects.
markdown: A single markdown document.
docx: A single Microsoft Word document.
epub,epub: A single epub ebook.
HTML_STYLE archwiki, github, custom CSS styles that loosely mimic the Arch Linux Wiki and GitHub documentation. See CUSTOM_CSS for more information about applying custom CSS styles.
PDF_MAIN_FONT Any installed font name. MkTechDocs will apply this font to the main “default” text of output if FORMAT is pdf or pdffloat.
PDF_MONO_FONT Any installed font name. MkTechDocs will apply this font to any sections of the PDF document that require a fixed-width font, such as code sections. Applies only to output if FORMAT is pdf or pdffloat.
TABLE_OF_CONTENTS_MAIN_DEPTH 1-6 The maximum number heading-level depth to count as a section number in the “main” document (not sub documents in htmlmulti format). For example, if TABLE_OF_CONTENTS_MAIN_DEPTH is 2, all level-three headings and greater will not appear in the table of contents.
TABLE_OF_CONTENTS_SUB_DEPTH 1-6 The maximum number heading-level depth to count as a section number in sub documents for the htmlmulti format.
SECTION_NUMBERS true,false Include section numbers in headings and tables of content.
TITLE_PAGE Any. No spaces or puncutation. See the title pages section for more information. Applies only to pdf, pdffloat, docx, epub, and epub3 output formats.
IMAGES Path to images directory – can be absolute or relative to project directory. See the Images section for more information. Applies to all formats.

4.2 Advanced configuration

More complex documents may require finer-grain control over how documents are rendered and stylized.

Variable Name Possible Values Effects
BUILD_SCRIPT Any See the Pre- and post-build activities section for for information.
CUSTOM_CSS Absolute or relative path to CSS file If you provide a file path here to a CSS file, MkTechDocs will copy that file into your *_pages output directory. To understand MkTechDocs CSS, see the $MKTECHDOCSHOME/lib/*.css files.
CUSTOM_TEMPLATE Absolute or relative path to pandoc template If you provide a file path here to a template file, MkTechDocs will use that template instead of one of the defaults. To understand MkTechDocs template files, see the $MKTECHDOCSHOME/*_template.html files. In addition, please see man pandoc -> “Variables set by pandoc” for a listing of pandoc variables available to template files. For more complex use of variables, see the Python Templates section.
KEEP_TEMP_FILES true,false If true, mktechdocs will not delete all temporary build files after completing a build.

5 Title pages

There are three ways to create a title page for your pdf, pdffloat, docx, epub, and epub3 document. In all three cases, you must create a standalone file and assign that filename to the TITLE_PAGE variable in your project’s configuration.

5.1 Title metadata block

% My Document Title
% John Doe; Jane Doe
% January 1, 2000

For more information about this syntax: man pandoc and search for “Metadata block.”

5.2 YAML metadata block

---
title: My Document Title
author:
  - John Doe
  - Jane Doe
date: January 1, 2001
---

5.3 Templates

If your title page requires dynamic content (e.g. non-static date, version number) you can create a Python template to output information as in the above two examples. Follow along with the example in Python templates section.

6 Useful Blocks

MkTechDocs contains built-in blocks that are useful when creating documentation.

6.1 Comment

Comment blocks are ignored by MkTechDocs.

E.g.

```comment
This text will be ignored.
```

6.2 Include

The include block was introduced in the basics section.

Parameter Description
heading-level Provides a hint to MkTechDocs about the current heading level. In order for MkTechDocs to produce nested headings, this parameter should always be used.

E.g.

# My header

```{.include heading-level=1}
myinclude.md
mysecondinclude.md
```

include blocks support infinte recursion, so included documents can contain included documents and so on.

6.3 Include-code

The include-code block is used to include source-code files with syntax highlighting.

Parameter Description
language Optional parameter that tells MkTechDocs what language the source code is. If no language is provided, MkTechDocs will use the file’s extension. E.g. file.c -> language=c

E.g.

```{.include-code language=c}
/home/mylogin/MkTechDocs/docs/file1.h
/home/mylogin/MkTechDocs/docs/file1.c
```

Produces:

#ifndef SOMETHING
typedef struct foo {
	int a;
} SOMETHING;
#endif
#include <stdio.h>
#include "file1.h"

int main(int argc, char **argv)
{
	printf("Size of SOMETHING is: %lu\n", sizeof(SOMETHING));
	return 0;
}

Note: Paths are an issue here. Because MkTechDocs projects are built inside your project directory, for files to be included correctly a full path or a path relative to the project directory must be given. MkTechDocs does its best to copy image files and directories to the output directory, but you may need to create a post-build script for more complex operations.

6.4 Note

note blocks are used to create documentation “notes” that appear highlighted in documentation text for HTML output formats. For PDF output, the content of the note block is prepended with “Note:”.

E.g.

```note
This text will appear inside its own highlighted block and be prepended with "Note:".
```

Produces:

Note: This text will appear inside its own highlighted block and be prepended with "Note:".

6.5 Plantuml

plantuml blocks let you include PlantUML diagramming UML directly in your documentation.

Parameter Description
title The title of the diagram. Used for a caption that appears beneath the diagram.

E.g.

# My Diagram

```{.plantuml title="My Diagram"}
A->B    
```

Produces:

My Diagram

Note: TBD: This is currently not working.

6.6 Tip

tip blocks are used to create documentation “tips” that appear highlighted in documentation text for HTML output formats. For PDF output, the content of the tip block is prepended with “Tip:”.

E.g.

```tip
This text will appear inside its own highlighted block and be prepended with "Tip:".
```

Produces:

Tip: This text will appear inside its own highlighted block and be prepended with "Tip:".

7 Document links

Pandoc automatically creates anchor names for every heading within your documents. Therefore, generating internal links is theoretically easy thanks to Pandoc’s naming convention. For example, suppose you create the following header:

# My New Header

Here’s how you’d normally create a link to it using Pandoc naming conventions:

Some markdown text that contains a [link](#my-new-header)

Because MkTechDocs can generate documents that require internal links that cross pages as well as documents that contain only internal relative links (as illustrated above), some thought has to be given to how links are created. When you generate internal links in your document, assume that your top-level included documents exist as HTML documents. For example, consider the following master document:

```{.include heading-level=0}
introduction.md
middle-section.md
end.md
```

To create links to other parts of your document, point your links to headings in introduction.html, middle-section.html, and end.html. For example, suppose introduction.md contained:

# Some heading

Some text

# Some other heading

Some more interesting text.

To refer to a section in this file in another file:

# Middle section

Here is a [link](introduction.html#some-heading) that points to another file.

If you follow this convention, MkTechDocs will produce correct links for every available format.

8 Python templates

MkTechDocs contains built-in support for Jinja2 templates. Using jinja2 templates is simply a matter of creating templates (plain text files containing Jinja2 markup code) and adding a .pyt extension, or, in the case of your header, landing page, and footer, a .htmlt extension, so MkTechDocs knows how to build them.

To use templates in MkTechDocs, you’ll need to create both a renderer and a template to render.

8.1 The template

A template is nothing more than plain text with some special Jinja2 markup thrown in. For example, let’s create a dynamic title page using templates. First, we’ll create a template that represents a dynamic YAML title block in our project directory:

title-page.pyt:

---
title: My Title {{ docVersion }}
author:
    - John Doe
    - Jane Doe
date: Built on {{ currentDateTime }}
---

The Jinja2 templating engine will replace the variables in the double curly brackets with values you provide in the renderer.

8.2 The renderer

Now, in your project directory, create a renderer with the same name as the template, but replace .pyt with .renderer. MkTechDocs will automatically use the renderer associated with the template name. Renderers are built with pure Python code. Continuing with our title-page example, here is a renderer that will properly render our title page template.

title-page.renderer:

#!/usr/bin/env python

import os
import sys
import datetime
from mktechdocslib import render_mktechdocs_jinja2_template

##
# Create a simple renderer function that ouputs a template passed in on the
# command line with the given variable dictionary.
#

def render():
  # Here, we hard code a document version, but in a real-world example, we'd
  # probably want to grab this from a configuration file or even a database.
  docVersion = "0.1a"

  # Now set up the date and time however you see fit
  currentDateTime = datetime.datetime.now().strftime("%I:%M%p on %B %d, %Y")

  varDictionary = {"docVersion":docVersion, "currentDateTime":currentDateTime}

  if not os.path.isfile(sys.argv[1]):
    sys.stderr.write("Cannot find " + sys.argv[1] + "\n")
    sys.exit(1)

  print render_mktechdocs_jinja2_template(sys.argv[1], varDictionary)

if __name__ == "__main__":
  render()

You can add as many variables to your varDictionary as necessary. You can even add entire modules and functions, if you need more logic in your templates.

Next, set the TITLE_PAGE variable in your project makefile to title-page.md. Why the .md extension? MkTeckDocs automatically converts anything with a .pyt extension to markdown before building your documents. Here’s a diagram that illustrates the process:

Template Rendering

Since title pages are only used in PDF, docx, and epub documents, you should also set FORMAT to pdf, pdffloat, docx, epub, or epub3.

8.3 Referencing templates in include blocks

MkTechDocs automatically detects *.pyt templates and converts them into markdown before processing include blocks. So, if you need to include the contents of a template’s output in another markdown file, simply treat the template as if it were markown. For example, if your template’s name is mytemplate.pyt reference the output in include blocks like:

```include
mytemplate.md
```

8.4 footer.html, header.html, and landing.html as templates

MkTechDocs will recognize footer.htmlt, header.htmlt, and landing.htmlt as templates and process into corresponding .html files, using similarly named renderers. E.g. footer.renderer.

8.5 Escaping curly brackets

Because every MkTechDocs document is ultimately converted into a large Python Jinja2 template, if you want to include double curly brackets as curly brackets in your document, for example, if you’re documenting Jinja2, you will need to escape them.

Here’s how:

    {% raw %}
 
    {{somevar}}
 
    {% endraw %}

If you need to escape a single curly bracket, you can use:

{{openCurlyBracket}} and {{closeCurlyBracket}}

8.6 For more information

For real-world examples of how to use Jinja2 templates with MkTechDocs, see the following files in the MkTechDocs directory:

docsbuild/runningfilters.pyt
docsbuild/runningfilters.renderer
docsbuild/scripts.pyt
docsbuild/scripts.renderer

These templates and renderers incorporate some pythondoc-like markup in the filters and scripts that come with MkTechDocs.

Jinja2 templates can be powerful documentation tools. Please reference the Jinja2 template documentation for more information 2>$template.err.

9 Managing document versions

For some documentation projects, managing document versions is vital. It is helpful if you define your document version in one place, so that any sub-documents that reference the version number get it from the same place.

The following is one reasonable way of doing this.

9.1 global_vars

Whether you’re using Python or Groovy, it’s probably a good idea to create a module (file) called global_vars.py to hold any static variables or functions you need to build your project, such as a version string. Here are the steps:

  1. Add your project’s directory to PYTHONPATH.
  2. For any documents that need access to information stored in global_vars, you simply convert the document into a template (by renaming it from .md to .pyt, for example).
  3. import global_vars as a module in your renderer and include it in your variable dictionary.
  4. Now, you can access global variables and functions in glob.py in your templates that have imported it: e.g. version=global_vars.VERS.

10 Incorporating Groovy code and templates

Groovy integration is an optional step and only necessary if you want to access existing Java libraries.

10.1 Installation of groovy components

First, start by examining and then running whichever install_deps_*_groovy.sh script in $MKTECHDOCSHOME/bin/groovy is appropriate for your architecture.

They install two packages, groovy and gradle. Then, they download a Groovy library called groovy-pandoc from GitHub and install the resulting jar file in $MKTECHDOCSHOME/lib.

10.1.1 Environment

Next, add $MKTECHDOCSHOME/bin/groovy to your PATH environment variable. Here’s an example in Bash:

export PATH=$PATH:$MKTECHDOCSHOME/bin/groovy

10.1.2 Testing the integration

Finally, run $MKTECHDOCSHOME/test/testgroovyintegration.sh. Your output should look something like this:

[~/MkTechDocs/test]$ testgroovyintegration.sh
Generating 'svg' plantuml diagram named 'Diagram'
Test dependency installation
============================

First, we'll try an include:

Include test file
-----------------

This text is coming from inside test\_install\_include.md.

Now, we'll try some plantUML:

![Diagram](./8410b9ad92ce7f3e2d564faff2b834b1.plantuml.svg){#myid}
Pandoc seems ok.


Checking groovy...

If this prints with no error, you are good to go
Groovy seems ok.
[~/MkTechDocs/test]$

10.2 Groovy templates

You can use Groovy templates (*.gt) as you would Jinja2 Python templates, except that Groovy templates do not require a renderer, which makes them somewhat more convenient. However, this comes at a price. Groovy templates are far slower than Jinja2 to generate, even with GroovyServ installed.

Groovy templates work like PHP or JSP files. You can include plain text markdown and Groovy clode blocks. Here’s an example:

# An example Groovy template

Let's count to 10:

<%
(1..10).each { n ->
    out.println(n)
}
%>

Here’s the output:

$ gtp testtemplate.gt
# An example Groovy template

Let's count to 10:

1
2
3
4
5
6
7
8
9
10


$

Notice that gtp is a standalone template renderer that MkTechDocs uses to render *.gt files.

10.3 Groovy classes

You might need standalone classes for your Groovy templates. To do this, create *.groovy files in your documentation directory. MkTechDocs will automatically detect and compile these into *.class files.

For example, MkTechDocs includes a class called MarkdownUtils, a utility class. Even though you wouldn’t likely use it in a real project, the following demonstrates the concept.

test.md:

# This is a level-one header

## This is a level-two header

mytempl.gt:

# Some heading

<%
import MarkdownUtils

def f = new File("./test.md")

out << MarkdownUtils.getMarkdownFileAndChangeHeader(f, 2)
%>

And here is the output:

[~/MkTechDocs/docs]$ gtp mytempl.gt
# Some heading

### This is a level-one header

#### This is a level-two header


[~/MkTechDocs/docs]$

Note how the heading levels were increased by two levels, which is exactly what the getMarkdownFileAndChangeHeader function does.

10.4 CLASSPATH

MkTechDocs creates a CLASSPATH automatically in order to process templates. However, if you need to add jars or directories to your CLASSPATH outside of MkTechDocs, simply create an environment variable. Here is an example in Bash:

export CLASSPATH=/path/to/my.jar

Now, when MkTechDocs runs, it will build a CLASSPATH that includes any existing CLASSPATH.

11 Team documentation

MkTechDocs is ideal for incorporating team-contributed documentation. Because MkTechDocs documentation is simple Markdown, team members can contribute or edit documents via Git (for example) and let a single technical writer manage the pull requests to keep things as simple and as clear as possible.

This makes scaling documentation easier because the technical writer is no longer limited to his or her own domain knowledge or learned knowledge.

That’s a simple case. The following describes a different scenario that lets agile developers contribute to documentation in a more specific and controlled way.

11.1 Case study

In this study, a number of different agile development teams are responsible for individual components of a larger system. A configuration repository maintained by the larger system consists of a number of YAML files that describe the general deployment of each component.

Here, components could be Docker containers, VMs, micro-services, or whatever. Each component could have a different but similar set of deployment instructions for each category below, or it could simply conform to some generic set of instructions and require no extra documentation.

  1. Overview
  2. Environmental-setup
  3. Pre-deployment
  4. Deployment
  5. Undeployment
  6. Upgrading
  7. Jenkins integration

Agile team members in this scenario are encouraged to contribute documentation in support of their respective components with those steps in mind. If they feel that specialized instructions for any of the above areas applies to their component, they should create a file in a provided directory within the documention tree that matches their component name in the configuration YAML. This file should contain a standalone Markdown document containing instructions specific for their component for that particular step. E.g. overview/mycomponent.md, upgrading/mycomponent.md.

For example, suppose a VM named vm-inventory exists in some YAML configuration file:

location:
  description: >
    This is a sample description of a particular location
  vm-deployments:
         .
         .
         .
    vm-inventory:
      vm-config: vmi.cfg
      release: 1.234
      notes: Some notes about this particular VM
        .
        .
        .

Now, we can create a template that loads the YAML containing the list of components, and looks for files with the same name in the directories listed above. If we find specialized instructions, we output them for a particular stage (e.g. Overview, Undeployment). If we find no specialized instructions, we output some generic ones.

11.2 Managing git commit messages

Often, it is useful for your document’s audience to understand what has changed from version to version. Rather than keeping track of this manually, in a “change log,” for example, you can leverage Git’s commit system.

The git log command shows a list of all commits to a respository. Each commit has a unique ID associated with it (SHA1 collisions not withstanding). This means that you can create “versions” by recording specific commit messages that represent the beginning and ending of a document version, pulling out all relevant commit messages dynamically when the document is built.

MkTechDocs includes this functionality in the mktechdocslib.py module. Here’s an example of how to use it.

11.2.1 GitCommitDB

What we’d like to do is create a simple database of Git messages that exist between two arbitrary Git commit IDs. First, we examine the output of git log to determine the first and last commit IDs that frame the version of the document we’re interested in creating. Note that the output of git log is ordered by commit date, so the most recent dates appear first.

Here is a sample Git log:

commit 20de5c8bf2a53efe54a71ae06e5b0b2a813d5176
Author: Seidel, Joseph (js2589) <spence@research.att.com>
Date:   Tue Mar 7 08:04:25 2017 -0500

    Fixed several formatting issues

commit 3523b7f36d928519bba9568852babd793e783a7c
Merge: cd6c4bc f9b3edf
Author: Seidel, Joseph (js2589) <spence@research.att.com>
Date:   Tue Mar 7 06:50:53 2017 -0600

    Merge pull request #71 in FOO_BAR/foo.bar.documentation from feature/FOOBAR/FOOBAR to master

    * commit 'f9b3edfb5194a2be0456d26d742043677948d5ea':
      Added some notes about documentation formats

commit f9b3edfb5194a2be0456d26d742043677948d5ea
Author: Seidel, Joseph (js2589) <spence@research.att.com>
Date:   Mon Mar 6 16:21:17 2017 -0500

    Added some notes about documentation formats

From this, we determine that the first commit ID is f9b3edfb5194a2be0456d26d742043677948d5ea and the last commit ID is 20de5c8bf2a53efe54a71ae06e5b0b2a813d5176.

Now, in a template.renderer, we can do something like this:

#!/usr/bin/env python

import os
import sys
import datetime
import global_vars
from mktechdocslib import GitCommitDB, render_mktechdocs_jinja2_template

##
# Create a simple renderer function that ouputs a template passed in on the
# command line with the given variable dictionary.
#

def render():
    messages = GitCommitDB("f9b3edfb5194a2be0456d26d742043677948d5ea", "20de5c8bf2a53efe54a71ae06e5b0b2a813d5176").messages
    varDictionary = {"gitMessages":messages,
                     "numCommits":len(messages)}
    
    if not os.path.isfile(sys.argv[1]):
        sys.stderr.write("Cannot find " + sys.argv[1] + "\n")
        sys.exit(1)

    print render_mktechdocs_jinja2_template(sys.argv[1], varDictionary)

if __name__ == "__main__":
    render()

Here, we import GitCommitDB from mktechdocslib and add the GitCommitDB.messages list object to our variable dictionary. messages contains a list of GitCommitMessage objects, which have the following properties:

Property Type Description
id String Commit ID
mergeID String Merge commit ID
author String Author of the commit
date String Commit date
message String Commit message
mergeCommit boolean True if the commit is a merge

Finally, here’s our template that displays the information we’re interested in:

# Document change log

{% if numCommits == 0 %}
Looks like no changes have been committed.
{% endif %}

{% for m in gitMessages %}

{% if not m.mergeCommit %}
**Date**: {{ m.date }}<br />
**Author**: {{ m.author }}<br />
**Message**: {{ m.message }}<br />
{% endif %}

{% endfor %}

The version in the example above was hard coded. In reality, it would be better to keep track of version information in a list of tuples. For example:

versions.py:

gVersions = [("1.0", "2017-04-06", "f9b3edfb5194a2be0456d26d742043677948d5ea", "20de5c8bf2a53efe54a71ae06e5b0b2a813d5176"),
             ("0.9", "2017-03-25", "d453519c024a49806fbd3a6740a9f0f586aaafbc", "57831131f186a60b864af27e53e06d2772a6d1ef")]

In this scheme, the latest version tuple would always be gVersions[0]. Or, if you prefer to store the versions in reverse order: gVersions[len(gVersions)-1]. Then, our renderer would look like this:

#!/usr/bin/env python

import os
import sys
import datetime
import global_vars
from versions import gVersions
from mktechdocslib import GitCommitDB, render_mktechdocs_jinja2_template

##
# Create a simple renderer function that ouputs a template passed in on the
# command line with the given variable dictionary.
#

def render():
    (versString, date, fcid, lcid) = gVersions[0]
    
    messages = GitCommitDB(fcid, lcid).messages
    
    varDictionary = {"gitMessages":messages,
                     "numCommits":len(messages),
                     "versString":versString,
                     "date":date}
    
    if not os.path.isfile(sys.argv[1]):
        sys.stderr.write("Cannot find " + sys.argv[1] + "\n")
        sys.exit(1)

    print render_mktechdocs_jinja2_template(sys.argv[1], varDictionary)

if __name__ == "__main__":
    render()

12 Install MkTechDocs

12.1 Docker

The easiest way to use MkTechDocs is via Docker.

docker run --rm --user $(id -u):$(id -g) -v $PROJECTDIR:/project jsseidel/mktechdocs

If $PROJECTDIR is an empty directory, MkTechDocs will create a new project. If $PROJECTDIR already contains a MkTechDocs project, MkTechDocs will build it.

Note: The Docker version of MkTechDocs does not support PDFs.

12.2 PPA

Add and install the Launchpad PPA in Ubuntu 18.04+:

sudo add-apt-repository -y ppa:jsseidel/mktechdocs
sudo apt update
sudo apt install -y mktechdocs

Then:

. /opt/mktechdocs/bin/mktechdocs.env

12.3 Deb package

Download the latest deb package from the MkTechDocs GitHub releases page: https://github.com/att/MkTechDocs/releases.

sudo apt install -y ./mktechdocs_18.04_1.0.8_amd64.deb

Then:

. /opt/mktechdocs/bin/mktechdocs.env

12.4 Source

To download the MkTechDocs source tree, clone the repo:

git clone https://github.com/att/MkTechDocs

Then, set up your environment.

13 Setting up your environment

Please export the following variables. Here is an example in Bash:

# MkTechDocs libraries and scripts require the following variable:
export MKTECHDOCSHOME=/path/to/cloned/repo/MkTechDocs

# Make sure the MkTechDocs bin directory is in your path. Here we add the current directory as well, `.`, because we want MkTechDocs to pick up templates in the project directories:
export PATH=$PATH:$MKTECHDOCSHOME/bin:.

# Let Python know where to find MkTechDocs libraries:
export PYTHONPATH=$MKTECHDOCSHOME/bin

If you installed MkTechDocs from a Debian package or Launchpad PPA, you can simply source the following to add these variable to your shell environment:

. /opt/mktechdocs/bin/mktechdocs.env

13.1 Running the dependencies installation script

Install MkTechDocs dependencies using the (admittedly imperfect) installation script. Consider the script more of a guide.

Examine the contents of the $MKTECHDOCSHOME/bin/install_deps*sh script appropriate for your architecture (Ubuntu, Arch, Fedora, and macOS are currently supported). For Groovy support, please also see the install_deps*sh scripts in the bin/groovy directory. The install script simply tries to install the various dependencies using an architecture-appropriate package manager and in some cases checks version numbers of installed binaries. You can also install the dependencies manually.

13.1.1 Dependencies

MkTechDocs requires the following:

Package Version Description
Bash any recent A shell.
Git any recent A distributed version-control system.
Pandoc >= 1.18 Pandoc is a powerful document conversion tool and is at the heart of MkTechDocs.
Make any recent A tool to help manage document dependencies during the build process.
Graphviz any recent A collection of open-source tools for drawing graphs in the DOT language.
Plantuml any recent A tool for creating UML diagrams that uses Graphviz underneath.
XeTeX any recent A suite of tools for building PDF documents.
Python 2.7 MkTechDocs is currently compatible with Python 2.7.
Jinja2 2+ A Python templating library.
homebrew** any recent A package manager for macOS.

** homebew: macOS only

Groovy/Java*

Package Version Description
Java* >= 1.8 MkTechDocs is currently compatible with Java 1.8
Groovy* > 2.0 A Java-like scripting language that extends Java.
groovy-pandoc* >= 0.8 A Groovy library used to build Pandoc filters for document transformation.
Gradle* > 3.0 A build automation system that is friendly to the Groovy language.

*: Java/Groovy dependencies are optional but ideal for interfacing MkTechDocs with existing Java libraries. Note that Groovy pandoc filters are orders of magnitude slower than their Python counterparts. To shorten build time, you might want to try GroovyServ.

13.2 Testing your environment

After installing the dependencies, run this:

cd $MKTECHDOCSHOME/test
./testinstall.sh

You should see something like this:

[~/MkTechDocs/test]$ ./testinstall.sh
Created image plantuml-c6117aebb4ce8e59ba2b46950eca869f.svg
Test dependency installation
============================

First, we'll try an include:

Include test file
-----------------

This text is coming from inside test\_install\_include.md.

Now, we'll try some plantUML:

![](plantuml-c6117aebb4ce8e59ba2b46950eca869f.svg)
Pandoc seems ok.


Checking python...

If this prints with no error, you are good to go!
Python seems ok.
[~/MkTechDocs/test]$

If you get errors, examine the output of the installation script to see where something went wrong.

13.3 A note about titlesec

MkTechDocs uses a TeX package named titlesec to do various things with PDF section numbers (e.g. 6.1.3.2). Unfortunately, Ubuntu 16.04 LTS ships with a buggy version of this package, which causes section numbers to be empty in various situations, so replacing it will be necessary if you plan to use section numbering in PDF documents. To replace it, first download the latest titlesec from https://www.ctan.org/tex-archive/macros/latex/contrib/titlesec.

Now:

unzip titlesec.zip
sudo rm -rf /usr/share/texlive/texmf-dist/tex/latex/titlesec
sudo mkdir /usr/share/texlive/texmf-dist/tex/latex/titlesec
sudo cp -r titlesec/* /usr/share/texlive/texmf-dist/tex/latex/titlesec/.

Section numbering should work as expected.

14 Running filters

Running MkTechDocs Pandoc filters outside of the normal build environment is easy if you’ve set up your environment properly:

pandoc --filter <flt-filtername.py> -f markdown -t some-other-format inputfile.md >outputfile.fmt

In some cases, MkTechDocs filters produce artifacts on STDERR. In these cases use something like the following:

pandoc --filter <flt-filtername.py> -f markdown -t some-other-format inputfile.md >outputfile.fmt 2>artifacts.out

The following Pandoc python filters come with MkTechDocs. Groovy versions are available as well in $MKTECHDOCSHOME/bin/groovy. They are run in the same way (but have a .groovy extension).

14.1 flt-comment-block.py

Adapted from https://github.com/jgm/pandocfilters/blob/master/examples/comments.py

Pandoc filter that causes everything between ‘<!– BEGIN COMMENT –>’ and ‘<!– END COMMENT –>’ to be ignored. The comment lines must appear on lines by themselves, with blank lines surrounding them.

E.g.

 <!-- BEGIN COMMENT -->

 This text will be ignored.

 <!-- END COMMENT -->

14.2 flt-comment.py

Ignores all text contained in the comment block. E.g.

 ```comment
 This text will be ignored.
 ````

14.3 flt-decrement-header-1.py

Decrements all headers by 1 level

14.4 flt-decrement-header-2.py

Decrements all headers by 2 levels

14.5 flt-decrement-header-3.py

Decrements all headers by 3 levels

14.6 flt-decrement-header-4.py

Decrements all headers by 4 levels

14.7 flt-decrement-header-5.py

Decrements all headers by 5 levels

14.8 flt-get-includes.py

Outputs all files referenced in any include blocks in a file to stderr. Used internally by MkTechDocs to build dependency lists.

14.9 flt-include-code.py

This filter allows you to include source-code files in your markdown.

You can provide one file per line. If you include the “language” property:

 ```{.include-code language="java"}
 myfile.java
 myfile.sh
 ```

The filter will use whatever language you specify in the property regardless of what the file extensions are, which you may or may not want.

If you provide something like the following:

 ```include-code
 myfile.java
 myfile.bash
 myfile.c
 ```

The filter will produce 3 code blocks with the language property automatically set based on the extension of the files within.

14.10 flt-include-doc-map.py

This filter is identical to the include filter, except that while building a document, it outputs a document map on stderr so that a script can figure out where each part of the document came from. E.g.

     ```include
     includethisfile.md
     ```

This filter is recursive, so you markdown can include other markdown to any level.

14.11 flt-include.py

This filter allows you to include other markdown in your markdown. E.g.

     ```include
     includethisfile.md
     ```

This filter is recursive, so your markdown can include other markdown to any level.

Header levels are automatically adjusted to the correct heading level if you use the “heading-level” attribute:

 # Heading One

 Some text

 ## Heading Two

 ```{.include heading-level=2}
 somefiletoinclude.md
 ````

“heading-level” represents the heading level where the include happens. On subsequent inclusion, the heading levels in the included file will be increased by two. Of course, you can always leave out the heading-level if you don’t want to alter the heading level of the included file.

Even multiple levels of include recursion will result in headers of the correct level. The reason this works is by virtue of that depth-first recursion.

The lowest level pages (i.e. the last file included in a recursive string of includes), are processed and stored to disk first.

When the next outer level is processed, those headers are adjusted and the previously adjusted levels in the included file are in effect adjusted again. Then, the next level is adjusted, and so are all the inclusions.

This results in the innermost include’s headers being adjusted upwards by however many levels of recursion there are.

14.12 flt-increment-header-1.py

Increments all headers by 1 level to a maximum of 6 levels.

14.13 flt-increment-header-2.py

Increments all headers by 2 levels to a maximum of 6 levels.

14.14 flt-increment-header-3.py

Increments all headers by 3 levels to a maximum of 6 levels.

14.15 flt-increment-header-4.py

Increments all headers by 4 levels to a maximum of 6 levels.

14.16 flt-increment-header-5.py

Increments all headers by 5 levels to a maximum of 6 levels.

14.17 flt-notetip.py

This filter replaces ‘note’ and ‘tip’ blocks with div tags of either a note or tip class

E.g.

 ```tip
 My tip
 ```

Becomes:

 <div class='tip'>**Tip**: My tip</div>

for markdown and html. For other output formats:

TIP: My tip

14.18 flt-plantuml.py

Adapted from: https://github.com/jgm/pandocfilters/blob/master/examples/plantuml.py

Turns PlantUML, e.g.:

 ```plantuml
 actor Foo1
 boundary Foo2
 control Foo3
 entity Foo4
 database Foo5
 Foo1 -> Foo2 : To boundary
 Foo1 -> Foo3 : To control
 Foo1 -> Foo4 : To entity
 Foo1 -> Foo5 : To database
 ````

Into an image for inclusion in HTML documents.

To affect the output format, provide a metadata variable on the command line:

 pandoc -M umlformat=[eps,svg] . . .

If no umlformat is indicated, the filter will default to svg.

Needs plantuml.jar from http://plantuml.com/.

14.19 flt-strip-pages-from-links.py

This filter turns links like this:

 [foo link](somepage.html#some-heading)

into:

 [foo link](#some-heading)

This is necessary to transform documents from multi-page to single page

14.20 flt-template.py

A starting point for building new filters.

15 Scripts

Several potentially useful scripts come with MkTechDocs.

15.1 dec-headers.sh

A utility script that decrements the headers in a given markdown file by a given number of levels.

A copy of the original file is kept using the origial file name and a PID as the extension.

Example usage dec-headers.sh file.md 1
Arguments 1 A file to alter.
2 The number of heading levels to decrement.
Special exit values None

15.2 escape-jinja-brackets.sh

This script escapes opening Jinja2 template brackets, {# to avoid internal build problems. Generally, you won’t need to run this script directly.

Example usage escape-jinja-brackets.sh file.md
Arguments 1 The file to alter
Special exit values None

15.3 inc-headers.sh

A utility script that increments the headers in a given markdown file by a given number of levels.

A copy of the original file is kept using the origial file name and a PID as the extension.

Example usage inc-headers.sh file.md 1
Arguments 1 A file to alter.
2 The number of heading levels to increment.
Special exit values None

15.4 install_deps_1804+_py.sh

This script will install all MkTechDocs dependencies in an Ubuntu 18.04 environment. It is not currently suited for using in a docker container or vagrantfile, although it would be easy enough to adapt it for such use.

Example usage install_deps_1804+_py.sh
Arguments None
Special exit values None

16 Addendum

16.1 Understanding the build process

The MkTechDocs build process is complex. The following is a step-by-step guide towards understanding how it works.

  1. MkTechDocs consists of a BASH control script and a library of Python and Groovy filters.
  2. The control script is responsible for importing the project configuration and calling pandoc with the correct arguments to build reasonably formatted documents in a variety of formats.
  3. The “magic” behind it all is pandoc ’s Abstract Syntax Tree (AST). Internally, pandoc converts all documents into AST, which can then be converted to JSON, which can then be manipulated by filters.
  4. MkTechDocs exploits this with a couple of libraries, GroovyPandoc, for Groovy template integration, and PandocFilters, for Python integration.
  5. Every time you run mktechdocs, the control script calls pandoc on your project’s master document. Some Bash scripting then takes care of some little details, which you can see for yourself by examining the script.

16.2 Contact

MkTechDocs was built and is maintained by Spencer Seidel.

17 Document change log

Date: Tue Feb 18 09:30:41 2020 -0500
Author: jsseidel
Message: Put docker builds and normal builds under same heading. [doc]

Date: Sat Apr 20 16:50:43 2019 -0400
Author: jsseidel
Message: Updated supported architectures section and reformatted a few paragraphs. [doc]

Date: Sat Apr 20 14:21:31 2019 -0400
Author: jsseidel
Message: Added section about how to install MkTechDocs from PPA, deb, and Docker. [doc]

Date: Fri Apr 19 15:14:08 2019 -0400
Author: jsseidel
Message: Updated to include better information about docker builds [doc]

Date: Fri Jan 25 16:35:25 2019 -0500
Author: jsseidel
Message: Added bit about putting the current directory in the path when setting up the environment to build. [doc]

Date: Thu Jan 17 11:12:22 2019 -0500
Author: jsseidel
Message: Fixed missing import in templates sample code [doc]

Date: Sat Sep 15 14:11:33 2018 -0400
Author: jsseidel
Message: Added a caveat about docker builds [doc]

Date: Sat Sep 15 13:53:53 2018 -0400
Author: jsseidel
Message: Updated build script docs. It was out of date. [doc]

Date: Fri Jul 6 10:31:15 2018 -0400
Author: jsseidel
Message: Added section about how to escape curly brackets [doc]

Date: Thu Jul 5 15:03:43 2018 -0400
Author: jsseidel
Message: Updated to point to tutorial MkTechDocs project [doc]

Date: Thu Jul 5 11:15:26 2018 -0400
Author: jsseidel
Message: Added section for doing docker builds [doc]

Date: Mon May 21 12:26:48 2018 -0400
Author: jsseidel
Message: Added a ‘doesn’t work’ note to the section about plantuml images titles. [doc]

Date: Mon May 21 11:41:47 2018 -0400
Author: jsseidel
Message: Added bit about 18.04 and cleaned up some wording [doc]

Date: Tue Apr 24 15:58:05 2018 -0400
Author: jsseidel
Message: Updated version to 1.0 and reset document change log [doc]

Date: Tue Apr 24 15:13:25 2018 -0400
Author: jsseidel
Message: Changed build system to remove makefile and updated documentation [doc]