Metadata-Version: 2.4
Name: remarshal
Version: 1.2.0
Summary: Convert between CBOR, JSON, MessagePack, TOML, and YAML
License: MIT
License-File: LICENSE
Keywords: converter,cbor,json,messagepack,msgpack,toml,yaml
Author: D. Bohdan
Author-email: dbohdan@dbohdan.com
Requires-Python: >=3.10
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: File Formats
Classifier: Topic :: File Formats :: JSON
Classifier: Topic :: Utilities
Provides-Extra: dev
Requires-Dist: cbor2 (>=5.6,<6.0)
Requires-Dist: codespell[toml] (>=2.2.5,<3.0) ; extra == "dev"
Requires-Dist: colorama (>=0.4.6,<0.5)
Requires-Dist: pyright (==1.1.399) ; extra == "dev"
Requires-Dist: pytest (>=8.0,<9.0) ; extra == "dev"
Requires-Dist: pytest-cov (>=4.1,<5.0) ; extra == "dev"
Requires-Dist: rich-argparse (>=1.4,<2.0)
Requires-Dist: ruamel.yaml (>=0.18.14,<0.19)
Requires-Dist: ruff (==0.11.5) ; extra == "dev"
Requires-Dist: snoop (>=0.6,<0.7) ; extra == "dev"
Requires-Dist: tomli (>=2.0.1,<3.0) ; extra == "dev"
Requires-Dist: tomli (>=2.0.1,<3.0) ; python_version < "3.11"
Requires-Dist: tomlkit (>=0.13.2,<0.14)
Requires-Dist: types-python-dateutil (>=2.8.19.13,<3.0) ; extra == "dev"
Requires-Dist: u-msgpack-python (>=2.8,<3.0)
Project-URL: Repository, https://github.com/remarshal-project/remarshal
Description-Content-Type: text/markdown

# Remarshal

Convert between CBOR, JSON, MessagePack, TOML, and YAML.
When installed,
Remarshal provides the command-line command `remarshal`
as well as the short commands
<code>{cbor,json,msgpack,toml,yaml}2<wbr>{cbor,json,msgpack,py,toml,yaml}</code>.
You can use these commands
to convert between formats,
reformat,
and detect errors.

Remarshal can also convert all supported formats to Python code.

## Known limitations and quirks

### YAML versions

Remarshal works with YAML 1.2 by default.
You can choose the input and/or output format `yaml-1.1` to work with YAML 1.1.
The input and output format `yaml-1.2` can be used to emphasize you mean YAML 1.2.
Conversion from YAML 1.1 to YAML 1.2 and from YAML 1.2 to YAML 1.1 is supported.

### Lossless by default; lossy must be enabled

Remarshal tries to convert documents without losing information by default.
This means that a document converted from format A to B and then back to A should be equal to the original document.
When a lossless conversion is impossible,
Remarshal exits with an error.

Use the command-line option `-k`/`--stringify` to relax this restriction.
It will make Remarshal do the following:

- When converting to JSON,   turn boolean and null keys and date-time keys and values into strings.
- When converting to TOML,
  turn boolean, date-time, and null keys and null values into strings.

This is **usually what you want**.
It isn't the default as a safeguard against information loss.

### Comments are removed

Remarshal does not preserve or convert TOML and YAML comments.

### Date-time conversion limitations

There are limitations
on what data can be converted
between what formats.

- CBOR, MessagePack, and YAML with binary fields cannot be converted
  to JSON or TOML.
  Binary fields can be converted between CBOR, MessagePack, and YAML.
- The following date-time value conversions are possible:
  - Local dates are converted between
    [CBOR RFC 8943](https://www.rfc-editor.org/rfc/rfc8943.html)
    dates (tag 1004),
    [TOML Local Dates](https://toml.io/en/v1.0.0#local-date),
    and
    [YAML timestamps](https://yaml.org/spec/1.2.2/#tags)
    without a time or a time zone.
  - Local date-time is converted between
    [TOML Local Date-Time](https://toml.io/en/v1.0.0#local-date-time)
    and
    [YAML timestamps](https://yaml.org/spec/1.2.2/#tags)
    without a time zone.
  - Date-time with a time zone
    is converted between
    [CBOR standard date-time strings](https://www.rfc-editor.org/rfc/rfc8949.html#stringdatetimesect)
    (tag 0),
    the
    [MessagePack Timestamp extension type](https://github.com/msgpack/msgpack/blob/master/spec.md#timestamp-extension-type),
    [TOML Offset Date-Times](https://toml.io/en/v1.0.0#offset-date-time),
    and
    [YAML timestamps](spec/1.2.2/#tags) with a time zone.
- [TOML Local Time](https://toml.io/en/v1.0.0#local-time)
  cannot be converted to a date-time in another format.
- All date-time types can be converted to JSON
  with the `-k`/`--stringify` option,
  which turns them into strings.
  Converting a document with a date-time type to JSON fails without `-k`/`--stringify`.

### Python output

Conversion to Python code is one-way.

The Python output is either from
[`repr`](https://docs.python.org/3/library/functions.html#repr)
(the default) or formatted by
[`pprint.pformat`](https://docs.python.org/3/library/pprint.html#pprint.pformat)
(when you pass the option `--indent`).
The default `repr` format ignores `-s`/`--sort-keys`.

The style of `pprint`
is probably not what you want your Python code to look like.
Apply your preferred Python formatter to it.

The Python output does not include the necessary `import` statements.
You may need to add `import datetime` before the data.

## Installation

You will need Python 3.10 or later.
Earlier versions of Python 3 will not work.

The recommended way to run Remarshal is to install the latest release
[from PyPI](https://pypi.org/project/remarshal/)
with
[pipx](https://github.com/pypa/pipx)
or
[uv](https://github.com/astral-sh/uv).

```sh
pipx install remarshal
# or
uv tool install remarshal
```

Regular installation is not mandatory.
The command `pipx run remarshal [arg ...]` will download Remarshal and run it from a temporary location.
It will cache the downloaded version for up to 14 days.
Remarshal will not be automatically upgraded during this period.
You can use `uvx remarshal [arg ...]` the same way.

It is also possible to install the current development version of Remarshal.
Prefer releases unless you have a reason to run a development version.

```sh
pipx install git+https://github.com/remarshal-project/remarshal
# or
uv tool install https://github.com/remarshal-project/remarshal
```

## Usage

```none
usage: remarshal [-h] [-v]
                 [-f {cbor,json,msgpack,toml,yaml,yaml-1.1,yaml-1.2}]
                 [-i <input>] [--indent <n>] [-k] [--max-values <n>]
                 [--multiline <n>] [-o <output>] [-s]
                 [-t {cbor,json,msgpack,python,toml,yaml,yaml-1.1,yaml-1.2}]
                 [--unwrap <key>] [--verbose] [--width <n>] [--wrap <key>]
                 [--yaml-style {,',",|,>}] [--yaml-style-newline {,',",|,>}]
                 [input] [output]

Convert between CBOR, JSON, MessagePack, TOML, and YAML.

positional arguments:
  input                 input file
  output                output file

options:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit
  -f, --from, --if, --input-format
{cbor,json,msgpack,toml,yaml,yaml-1.1,yaml-1.2}
                        input format
  -i, --input <input>   input file
  --indent <n>          JSON and YAML indentation
  -k, --stringify       turn into strings: boolean and null keys and date-time
                        keys and values for JSON; boolean, date-time, and null
                        keys and null values for TOML
  --max-values <n>      maximum number of values in input data (default
                        1000000, negative for unlimited)
  --multiline <n>       minimum number of items to make non-nested TOML array
                        multiline (default 6)
  -o, --output <output>
                        output file
  -s, --sort-keys       sort JSON, Python, and TOML keys instead of preserving
                        key order
  -t, --to, --of, --output-format
{cbor,json,msgpack,python,toml,yaml,yaml-1.1,yaml-1.2}
                        output format
  --unwrap <key>        only output the data stored under the given key
  --verbose             print debug information when an error occurs
  --width <n>           Python line width and YAML line width for long strings
                        (integer or 'inf')
  --wrap <key>          wrap the data in a map type with the given key
  --yaml-style {,',",|,>}
                        YAML formatting style
  --yaml-style-newline {,',",|,>}
                        YAML formatting style override for strings that
                        contain a newline
```

Instead of `remarshal` with format arguments,
you can use a short command
<code>{cbor,json,msgpack,toml,yaml}2<wbr>{cbor,json,msgpack,py,toml,yaml}</code>.
The `remarshal` command and the short commands
exit with status 0 on success,
1 on operational failure,
and 2 on failure to parse the command line.

If no input argument `input`/`-i input` is given or its value is `-`,
Remarshal reads input data from standard input.
Similarly,
with no `output`/`-o output` or an output argument that is `-`,
Remarshal writes the result to standard output.

### Wrappers

The options `--wrap` and `--unwrap` are available
to solve the problem of converting CBOR, JSON, MessagePack, and YAML data to TOML
when the top-level element of the data is not of a dictionary type
(i.e., not a map in CBOR and MessagePack,
an object in JSON,
or an associative array in YAML).
You cannot represent such data as TOML directly;
the data must be wrapped in a dictionary first.
Passing the option `--wrap some-key` to `remarshal` or one of its short commands
wraps the input data in a "wrapper" dictionary with one key, `some-key`,
with the input data as its value.
The option `--unwrap some-key` does the opposite:
it converts to the target format and outputs
only the value stored under the key `some-key`
in the top-level dictionary element of the input data;
the rest of the input is discarded.
If the top-level element is not a dictionary or does not have the key `some-key`,
`--unwrap some-key` causes an error.

The following shell transcript demonstrates the problem
and how `--wrap` and `--unwrap` solve it:

```
$ echo '[{"a":"b"},{"c":[1,2,3]}]' | remarshal --from json --to toml
Error: cannot convert non-dictionary data to TOML; use "--wrap" to wrap it in a dictionary

$ echo '[{"a":"b"},{"c":[1,2,3]}]' \
  | remarshal --from json --to toml --wrap main
[[main]]
a = "b"

[[main]]
c = [1, 2, 3]

$ echo '[{"a":"b"},{"c":[1,2,3]}]' \
  | remarshal --from json --wrap main - test.toml

$ remarshal test.toml --to json
{"main":[{"a":"b"},{"c":[1,2,3]}]}

$ remarshal test.toml --to json --unwrap main
[{"a":"b"},{"c":[1,2,3]}]
```

## Shell completions

Remarshal provides shell-completion files for Bash and fish in the directory [`completions/`](completions/).
You can install fish completions automatically by running `install.fish`.
You will need to install Bash completions manually.

## Examples

### TOML to YAML

```
$ remarshal example.toml --to yaml
title: TOML Example
owner:
  name: Tom Preston-Werner
  organization: GitHub
  bio: "GitHub Cofounder & CEO\nLikes tater tots and beer."
  dob: 1979-05-27 07:32:00+00:00
database:
  server: 192.168.1.1
  ports:
  - 8001
  - 8001
  - 8002
  connection_max: 5000
  enabled: true
servers:
  alpha:
    ip: 10.0.0.1
    dc: eqdc10
  beta:
    ip: 10.0.0.2
    dc: eqdc10
    country: 中国
clients:
  data:
  - - gamma
    - delta
  - - 1
    - 2
  hosts:
  - alpha
  - omega
products:
- name: Hammer
  sku: 738594937
- name: Nail
  sku: 284758393
  color: gray
```

### JSON to TOML

```
$ curl -f 'https://archive-api.open-meteo.com/v1/era5?latitude=50.43&longitude=30.52&start_date=2014-10-05&end_date=2014-10-05&hourly=temperature_2m' \
  | remarshal --from json --to toml \
  ;
latitude = 50.439365
longitude = 30.476192
generationtime_ms = 0.03254413604736328
utc_offset_seconds = 0
timezone = "GMT"
timezone_abbreviation = "GMT"
elevation = 147.0

[hourly_units]
time = "iso8601"
temperature_2m = "°C"

[hourly]
time = [
    "2014-10-05T00:00",
    "2014-10-05T01:00",
    "2014-10-05T02:00",
    "2014-10-05T03:00",
    "2014-10-05T04:00",
    "2014-10-05T05:00",
    "2014-10-05T06:00",
    "2014-10-05T07:00",
    "2014-10-05T08:00",
    "2014-10-05T09:00",
    "2014-10-05T10:00",
    "2014-10-05T11:00",
    "2014-10-05T12:00",
    "2014-10-05T13:00",
    "2014-10-05T14:00",
    "2014-10-05T15:00",
    "2014-10-05T16:00",
    "2014-10-05T17:00",
    "2014-10-05T18:00",
    "2014-10-05T19:00",
    "2014-10-05T20:00",
    "2014-10-05T21:00",
    "2014-10-05T22:00",
    "2014-10-05T23:00",
]
temperature_2m = [
    5.7,
    5.3,
    5.0,
    4.8,
    4.6,
    4.6,
    7.0,
    8.9,
    10.8,
    12.2,
    13.3,
    13.9,
    13.9,
    13.7,
    13.3,
    12.3,
    11.1,
    10.2,
    9.4,
    8.5,
    8.2,
    7.9,
    8.0,
    7.8,
]
```

Remarshal controls the number of items at which a TOML array becomes multiline,
but it does not control the line width.
You can use
[`taplo fmt`](https://taplo.tamasfe.dev/cli/usage/formatting.html)
for finer TOML formatting.

## License

MIT.
See the file
[`LICENSE`](LICENSE).

`example.toml` from <https://github.com/toml-lang/toml>.
`example.cbor`,
`example.json`,
`example.msgpack`,
`example.py`,
`example.yml`,
`tests/bin.msgpack`,
and `tests/bin.yml`
are derived from it.

