aboutsummaryrefslogtreecommitdiffstats
path: root/scripts/json_overview_image_info.py
Commit message (Collapse)AuthorAgeFilesLines
* build,json: fix compatibility with Python 3.5Paul Spooren2020-07-131-1/+1
| | | | | | | | The f-string feature was introduced in Python 3.6. As Buildbots may run on Debian 9, which comes per default with Python 3.5, this would cause an issue. Instead of f-strings use the *legacy* `.format()` function. Signed-off-by: Paul Spooren <mail@aparcar.org>
* build: fix compatibility with python 3.6Huangbin Zhan2020-07-071-3/+4
| | | | | | | | On a system python3 is linked to python3.6, fail to perform json_overview_image_info and got `TypeError: __init__() got an unexpected keyword argument 'capture_output'`. This patch emulate the behaviour on python 3.7+. Signed-off-by: Huangbin Zhan <zhanhb88@gmail.com>
* build,json: fix build failure in case no data is foundDaniel Golle2020-07-031-18/+16
| | | | | | | | Only collect arch_packages if actually generating any output. Fixes: commit f09b9319 ("build,json: store arch_packages in profiles.json"( Signed-off-by: Daniel Golle <daniel@makrotopia.org> Signed-off-by: Paul Spooren <mail@aparcar.org>
* build,json: store arch_packages in profiles.jsonPaul Spooren2020-07-031-2/+5
| | | | | | | | | The `arch_packages` contains the supported package architecture. Previously it was necessary to parse the `Packages` index for the line `Architecture:`, requiring both an additional parser and file download. Signed-off-by: Paul Spooren <mail@aparcar.org>
* build: store default/device packages in JSONPaul Spooren2020-06-301-2/+18
| | | | | | | | | | | | With this commit the `profiles.json` contain both the target specific `default_packages` as well as the device specific `device_packages` as a array of strings. This information is required for downstream projects like the various web-based interactive firmware generators. Signed-off-by: Daniel Golle <daniel@makrotopia.org> Signed-off-by: Paul Spooren <mail@aparcar.org>
* scripts: JSON merge don't crash if no JSON foundPaul Spooren2020-04-141-2/+0
| | | | | | | | | | | | | | | | | | The JSON `WORK_DIR` ($(KDIR)/json_info_files) is only created if the new image generation methods from `image.mk` are used. However some targets like `armvirt` do not use it yet, so the folder is never created. The `json_overview_image_info.py` script used to raise an error if the given `WORK_DIR` isn't a folder, however it should just notify about missing JSON files. This patch removes the Python assert and exists with code 0 even if no JSON files were found, as this is not necessarily an error but simply not yet implemented. Using `glob` on an not existing `Path` results in an empty list, therefore the for loop won't run. Signed-off-by: Paul Spooren <mail@aparcar.org> CC: Petr Štetiar <ynezz@true.cz>
* build: refactor JSON info files to `profiles.json`Paul Spooren2020-04-031-0/+39
JSON info files contain machine readable information of built profiles and resulting images. These files were added in commit 881ed09ee6e2 ("build: create JSON files containing image info"). They are useful for firmware wizards and script checking for reproducibility. Currently all JSON files are stored next to the built images, resulting in up to 168 individual files for the ath79/generic target. This patch refactors the JSON creation to store individual per image (not per profile) files in $(BUILD_DIR)/json_info_files and create an single overview file called `profiles.json` in the target directory. Storing per image files and not per profile solves the problem of parallel file writes. If a profiles sysupgrade and factory image are finished at the same time both processes would write to the same JSON file, resulting in randomly broken outputs. Some target like x86/64 do not use the image code yet, resulting in missing JSON files. If no JSON info files were created, no `profiles.json` files is created as it would be empty anyway. As before, this creation is enabled by default only if `BUILDBOT` is set. Tested via buildroot & ImageBuilder on ath79/generic, imx6 and x86/64. Signed-off-by: Paul Spooren <mail@aparcar.org> [json_info_files dir handling in Make, if case refactoring] Signed-off-by: Petr Štetiar <ynezz@true.cz>
xception */ .highlight .nf { color: #0066bb; font-weight: bold } /* Name.Function */ .highlight .nl { color: #336699; font-style: italic } /* Name.Label */ .highlight .nn { color: #bb0066; font-weight: bold } /* Name.Namespace */ .highlight .py { color: #336699; font-weight: bold } /* Name.Property */ .highlight .nt { color: #bb0066; font-weight: bold } /* Name.Tag */ .highlight .nv { color: #336699 } /* Name.Variable */ .highlight .ow { color: #008800 } /* Operator.Word */ .highlight .w { color: #bbbbbb } /* Text.Whitespace */ .highlight .mb { color: #0000DD; font-weight: bold } /* Literal.Number.Bin */ .highlight .mf { color: #0000DD; font-weight: bold } /* Literal.Number.Float */ .highlight .mh { color: #0000DD; font-weight: bold } /* Literal.Number.Hex */ .highlight .mi { color: #0000DD; font-weight: bold } /* Literal.Number.Integer */ .highlight .mo { color: #0000DD; font-weight: bold } /* Literal.Number.Oct */ .highlight .sa { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Affix */ .highlight .sb { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Backtick */ .highlight .sc { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Char */ .highlight .dl { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Delimiter */ .highlight .sd { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Doc */ .highlight .s2 { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Double */ .highlight .se { color: #0044dd; background-color: #fff0f0 } /* Literal.String.Escape */ .highlight .sh { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Heredoc */ .highlight .si { color: #3333bb; background-color: #fff0f0 } /* Literal.String.Interpol */ .highlight .sx { color: #22bb22; background-color: #f0fff0 } /* Literal.String.Other */ .highlight .sr { color: #008800; background-color: #fff0ff } /* Literal.String.Regex */ .highlight .s1 { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Single */ .highlight .ss { color: #aa6600; background-color: #fff0f0 } /* Literal.String.Symbol */ .highlight .bp { color: #003388 } /* Name.Builtin.Pseudo */ .highlight .fm { color: #0066bb; font-weight: bold } /* Name.Function.Magic */ .highlight .vc { color: #336699 } /* Name.Variable.Class */ .highlight .vg { color: #dd7700 } /* Name.Variable.Global */ .highlight .vi { color: #3333bb } /* Name.Variable.Instance */ .highlight .vm { color: #336699 } /* Name.Variable.Magic */ .highlight .il { color: #0000DD; font-weight: bold } /* Literal.Number.Integer.Long */
# Flashing Instructions and Bootloader Information

There are quite a few different types of bootloaders that keyboards use, and just about all of them use a different flashing method. Luckily, projects like the [QMK Toolbox](https://github.com/qmk/qmk_toolbox/releases) aim to be compatible with all the different types without having to think about it much, but this article will describe the different types of bootloaders, and available methods for flashing them.

If you have a bootloader selected with the `BOOTLOADER` variable in your `rules.mk`, QMK will automatically calculate if your .hex file is the right size to be flashed to the device, and output the total size in bytes (along with the max).

## DFU

Atmel's DFU bootloader comes on all atmega32u4 chips by default, and is used by many keyboards that have their own ICs on their PCBs (Older OLKB boards, Clueboards). Some keyboards may also use LUFA's DFU bootloader (or QMK's fork) (Newer OLKB boards) that adds in additional features specific to that hardware.

To ensure compatibility with the DFU bootloader, make sure this block is present your `rules.mk` (optionally with `lufa-dfu` or `qmk-dfu` instead):

```make
# Bootloader selection
#   Teensy       halfkay
#   Pro Micro    caterina
#   Atmel DFU    atmel-dfu
#   LUFA DFU     lufa-dfu
#   QMK DFU      qmk-dfu
#   ATmega32A    bootloadHID
#   ATmega328P   USBasp
BOOTLOADER = atmel-dfu
```

Compatible flashers:

* [QMK Toolbox](https://github.com/qmk/qmk_toolbox/releases) (recommended GUI)
* [dfu-programmer](https://github.com/dfu-programmer/dfu-programmer) / `:dfu` in QMK (recommended command line)
* [Atmel's Flip](http://www.microchip.com/developmenttools/productdetails.aspx?partno=flip) (not recommended)

Flashing sequence:

1. Press the `RESET` keycode, or tap the RESET button (or short RST to GND).
2. Wait for the OS to detect the device
3. Erase the memory (may be done automatically)
4. Flash a .hex file
5. Reset the device into application mode (may be done automatically)

or:

    make <keyboard>:<keymap>:dfu

### QMK DFU

QMK has a fork of the LUFA DFU bootloader that allows for a simple matrix scan for exiting the bootloader and returning to the application, as well as flashing an LED/making a ticking noise with a speaker when things are happening. To enable these features, use this block in your `config.h` (The key that exits the bootloader needs to be hooked-up to the INPUT and OUTPUT defined here):

    #define QMK_ESC_OUTPUT F1 // usually COL
    #define QMK_ESC_INPUT D5 // usually ROW
    #define QMK_LED E6
    #define QMK_SPEAKER C6

The Manufacturer and Product names are automatically pulled from your `config.h`, and "Bootloader" is added to the product.

To generate this bootloader, use the `bootloader` target, eg `make planck/rev4:default:bootloader`.

To generate a production-ready .hex file (containing the application and the bootloader), use the `production` target, eg `make planck/rev4:default:production`.

### DFU commands

There are a number of DFU commands that you can use to flash firmware to a DFU device:

* `:dfu` - This is the normal option and waits until a DFU device is available, and then flashes the firmware. This will check every 5 seconds, to see if a DFU device has appeared.
* `:dfu-ee` - This flashes an `eep` file instead of the normal hex.  This is uncommon. 
* `:dfu-split-left` - This flashes the normal firmware, just like the default option (`:dfu`). However, this also flashes the "Left Side" EEPROM file for split keyboards. _This is ideal for Elite C based split keyboards._
* `:dfu-split-right` - This flashes the normal firmware, just like the default option (`:dfu`). However, this also flashes the "Right Side" EEPROM file for split keyboards. _This is ideal for Elite C based split keyboards._

## Caterina

Arduino boards and their clones use the [Caterina bootloader](https://github.com/arduino/ArduinoCore-avr/tree/master/bootloaders/caterina) (any keyboard built with a Pro Micro, or clone), and uses the avr109 protocol to communicate through virtual serial. Bootloaders like [A-Star](https://www.pololu.com/docs/0J61/9) are based on Caterina.

To ensure compatibility with the Caterina bootloader, make sure this block is present your `rules.mk`:

```make
# Bootloader selection
#   Teensy       halfkay
#   Pro Micro    caterina
#   Atmel DFU    atmel-dfu
#   LUFA DFU     lufa-dfu
#   QMK DFU      qmk-dfu
#   ATmega32A    bootloadHID
#   ATmega328P   USBasp
BOOTLOADER = caterina
```

Compatible flashers:

* [QMK Toolbox](https://github.com/qmk/qmk_toolbox/releases) (recommended GUI)
* [avrdude](http://www.nongnu.org/avrdude/) with avr109 / `:avrdude` (recommended command line)
* [AVRDUDESS](https://github.com/zkemble/AVRDUDESS)

Flashing sequence:

1. Press the `RESET` keycode, or short RST to GND quickly (you only have 7 seconds to flash once it enters)
2. Wait for the OS to detect the device
3. Flash a .hex file
4. Wait for the device to reset automatically

or

    make <keyboard>:<keymap>:avrdude


### Caterina commands

There are a number of DFU commands that you can use to flash firmware to a DFU device:

* `:avrdude` - This is the normal option which waits until a Caterina device is available (by detecting a new COM port), and then flashes the firmware.
* `:avrdude-loop` - This runs the same command as `:avrdude`, but after each device is flashed, it will attempt to flash again.  This is useful for bulk flashing. _This requires you to manually escape the loop by hitting Ctrl+C._
* `:avrdude-split-left` - This flashes the normal firmware, just like the default option (`:avrdude`). However, this also flashes the "Left Side" EEPROM file for split keyboards. _This is ideal for Pro Micro based split keyboards._
* `:avrdude-split-right` - This flashes the normal firmware, just like the default option (`:avrdude`). However, this also flashes the "Right Side" EEPROM file for split keyboards. _This is ideal for Pro Micro based split keyboards._



## Halfkay

Halfkay is a super-slim protocol developed by PJRC that uses HID, and comes on all Teensys (namely the 2.0).

To ensure compatibility with the Halfkay bootloader, make sure this block is present your `rules.mk`:

```make
# Bootloader selection
#   Teensy       halfkay
#   Pro Micro    caterina
#   Atmel DFU    atmel-dfu
#   LUFA DFU     lufa-dfu
#   QMK DFU      qmk-dfu
#   ATmega32A    bootloadHID
#   ATmega328P   USBasp
BOOTLOADER = halfkay
```

Compatible flashers:

* [QMK Toolbox](https://github.com/qmk/qmk_toolbox/releases) (recommended GUI)
* [Teensy Loader](https://www.pjrc.com/teensy/loader.html)
* [Teensy Loader Command Line](https://www.pjrc.com/teensy/loader_cli.html) (recommended command line)

Flashing sequence:

1. Press the `RESET` keycode, or short RST to GND quickly (you only have 7 seconds to flash once it enters)
2. Wait for the OS to detect the device
3. Flash a .hex file
4. Reset the device into application mode (may be done automatically)

## USBasploader

USBasploader is a bootloader developed by matrixstorm. It is used in some non-USB AVR chips such as the ATmega328P, which run V-USB.

To ensure compatibility with the USBasploader bootloader, make sure this block is present in your `rules.mk`:

```make
# Bootloader selection
#   Teensy       halfkay
#   Pro Micro    caterina
#   Atmel DFU    atmel-dfu
#   LUFA DFU     lufa-dfu
#   QMK DFU      qmk-dfu
#   ATmega32A    bootloadHID
#   ATmega328P   USBasp
BOOTLOADER = USBasp
```

Compatible flashers:

* [QMK Toolbox](https://github.com/qmk/qmk_toolbox/releases) (recommended GUI)
* [avrdude](http://www.nongnu.org/avrdude/) with the `usbasp` programmer
* [AVRDUDESS](https://github.com/zkemble/AVRDUDESS)

Flashing sequence:

1. Press the `RESET` keycode, or keep the boot pin shorted to GND while quickly shorting RST to GND
2. Wait for the OS to detect the device
3. Flash a .hex file
4. Reset the device into application mode (may be done automatically)

## BootloadHID

BootloadHID is a USB bootloader for AVR microcontrollers. The uploader tool requires no kernel level driver on Windows and can therefore be run without installing any DLLs.

To ensure compatibility with the bootloadHID bootloader, make sure this block is present your `rules.mk`:

```make
# Bootloader selection
#   Teensy       halfkay
#   Pro Micro    caterina
#   Atmel DFU    atmel-dfu
#   LUFA DFU     lufa-dfu
#   QMK DFU      qmk-dfu
#   ATmega32A    bootloadHID
#   ATmega328P   USBasp
BOOTLOADER = bootloadHID
```

Compatible flashers:

* [HIDBootFlash](http://vusb.wikidot.com/project:hidbootflash) (recommended Windows GUI)
* [bootloadhid Command Line](https://www.obdev.at/products/vusb/bootloadhid.html) / `:BootloadHID` in QMK (recommended command line)

Flashing sequence:

1. Enter the bootloader using any of the following methods:
    * Tap the `RESET` keycode (may not work on all devices)
    * Hold the salt key while plugging the keyboard in (usually documented within keyboard readme) 
2. Wait for the OS to detect the device
3. Flash a .hex file
4. Reset the device into application mode (may be done automatically)

or:

    make <keyboard>:<keymap>:bootloadHID

## STM32

All STM32 chips come preloaded with a factory bootloader that cannot be modified nor deleted. Some STM32 chips have bootloaders that do not come with USB programming (e.g. STM32F103) but the process is still the same.

At the moment, no `BOOTLOADER` variable is needed on `rules.mk` for STM32.

Compatible flashers:

* [QMK Toolbox](https://github.com/qmk/qmk_toolbox/releases) (recommended GUI)
* [dfu-util](https://github.com/Stefan-Schmidt/dfu-util) / `:dfu-util` (recommended command line)

Flashing sequence:

1. Enter the bootloader using any of the following methods:
    * Tap the `RESET` keycode (may not work on STM32F042 devices)
    * If a reset circuit is present, tap the RESET button
    * Otherwise, you need to bridge BOOT0 to VCC (via BOOT0 button or bridge), short RESET to GND (via RESET button or bridge), and then let go of the BOOT0 bridge
2. Wait for the OS to detect the device
3. Flash a .bin file
    * You will receive a warning about the DFU signature; Just ignore it
4. Reset the device into application mode (may be done automatically)
    * If you are building from command line (e.g. `make planck/rev6:default:dfu-util`), make sure that `:leave` is passed to the `DFU_ARGS` variable inside your `rules.mk` (e.g. `DFU_ARGS = -d 0483:df11 -a 0 -s 0x08000000:leave`) so that your device resets after flashing

### STM32 Commands

There are a number of DFU commands that you can use to flash firmware to a STM32 device:

* `:dfu-util` - The default command for flashing to STM32 devices, and will wait until an STM32 bootloader device is present.
* `:dfu-util-split-left` - This flashes the normal firmware, just like the default option (`:dfu-util`). However, this also configures the "Left Side" EEPROM setting for split keyboards.
* `:dfu-util-split-right` - This flashes the normal firmware, just like the default option (`:dfu-util`). However, this also configures the "Right Side" EEPROM setting for split keyboards.
* `:st-link-cli` - This allows you to flash the firmware via ST-LINK's CLI utility, rather than dfu-util.