Split core functionality and support orjson and msgspec (#9)

## Summary of changes

### Refactor common functionality into base module
This allows support multiple JSON encoders by having common
functionality in `pythonjsonlogger.core` and then specialist formatters
for each encoder.

This is useful / needed, as not all JSON encoders support the
`json.dumps` or `json.JSONEncoder` interfaces exactly. This enables us
to support other JSON encoders like orjson and msgspec. In the future we
may add support for other encoders.

### Better handling for custom styles
Achieved by mimicking `logging.Formatter.__init__` without actually
calling it.

A code snippet is worth `2**10` words:

```python
from pythonjsonlogger.core import BaseJsonLogger

class CommaSupport(BaseJsonFormatter):
    def parse(self) -> list[str]:
        if isinstance(self._style, str) and self._style == ",":
            return self._fmt.split(",")
        return super().parse()

f = CommaSupport("message,asctime", style=",", validate=False)
```

### Rename `jsonlogger` module to `json` module

Compatibility is maintained for the moment using `__getattr__` in
`__init__`.

This is to enable more consistent naming of implementation specific
module names. It also stops throwing around the word "logger" when this
module only contains formatters.

### Add support for orjson
[orjson](https://github.com/ijl/orjson) is a high performance (and more
JSON spec correct) encoder. Given how many logging calls may occur -
having a performant formatter available is important.

This includes ensuring it is covered in tests on appropriate platforms.

Note: orjson is not supported on pypy, and currently does not build for
py313.

### Add support for msgspec
[msgspec](https://jcristharif.com/msgspec/index.html) is another library
containing a high performance JSON encoder.

Note: msgspec is not supported on pypy, and currently does not build for
py313.

### Drops python 3.7 support
This is primary due do making use of the
[`validate`](https://docs.python.org/3/library/logging.html#formatter-objects)
argument. I was also having issues with CI because python 3.7 is not support on most "latest"

Author: nhairs

.github/workflows/test-suite.yml

@@ -33,18 +33,18 @@ jobs:
     needs: [lint]
     runs-on: "${{ matrix.os }}"
     strategy:
+      fail-fast: false  # allow tests to run on all platforms
       matrix:
         python-version:
-          - "pypy-3.7"
           - "pypy-3.8"
           - "pypy-3.9"
           - "pypy-3.10"
-          - "3.7"
           - "3.8"
           - "3.9"
           - "3.10"
           - "3.11"
           - "3.12"
+          - "3.13-dev"
         os:
           - ubuntu-latest
           - windows-latest

CHANGELOG.md

@@ -4,6 +4,33 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [3.1.0.rc1](https://github.com/nhairs/python-json-logger/compare/v3.0.1...v3.1.0.rc1) - 2023-05-03
+
+This splits common funcitonality out to allow supporting other JSON encoders. Although this is a large refactor, backwards compatibility has been maintained.
+
+### Added
+- `.core` - more details below.
+- Orjson encoder support via `.orjson.OrjsonFormatter`.
+- MsgSpec encoder support via `.msgspec.MsgspecFormatter`.
+
+### Changed
+- `.jsonlogger` has been moved to `.json` with core functionality moved to `.core`.
+- `.core.BaseJsonFormatter` properly supports all `logging.Formatter` arguments:
+  - `fmt` is unchanged.
+  - `datefmt` is unchanged.
+  - `style` can now support non-standard arguments by setting `validate` to `False`
+  - `validate` allows non-standard `style` arguments or prevents calling `validate` on standard `style` arguments.
+  - `default` is ignored.
+
+### Deprecated
+- `.jsonlogger` is now `.json`
+- `.jsonlogger.RESERVED_ATTRS` is now `.core.RESERVED_ATTRS`.
+- `.jsonlogger.merge_record_extra` is now `.core.merge_record_extra`.
+
+### Removed
+- Python 3.7 support dropped
+- `.jsonlogger.JsonFormatter._str_to_fn` replaced with `.core.str_to_object`.
+
 ## [3.0.1](https://github.com/nhairs/python-json-logger/compare/v3.0.0...v3.0.1) - 2023-04-01
 
 ### Fixes

README.md

@@ -26,7 +26,7 @@ Until the PEP 541 request is complete you will need to install directly from git
 To install from releases:
 
 ```shell
-# 3.0.0 wheel
+# e.g. 3.0.0 wheel
 pip install 'python-json-logger@https://github.com/nhairs/python-json-logger/releases/download/v3.0.0/python_json_logger-3.0.0-py3-none-any.whl'
 ```
 
@@ -53,38 +53,30 @@ pip install -e .
 
 ## Usage
 
+Python JSON Logger provides `logging.Formatter`s that encode the logged message into JSON. Although a variety of JSON encoders are supported, in the following examples we will use the `pythonjsonlogger.json.JsonFormatter` which uses the the `json` module from the standard library.
+
 ### Integrating with Python's logging framework
 
-Json outputs are provided by the JsonFormatter logging formatter. You can add the custom formatter like below:
+To produce JSON output, attach the formatter to a logging handler:
 
 ```python
     import logging
-    from pythonjsonlogger import jsonlogger
+    from pythonjsonlogger.json import JsonFormatter
 
     logger = logging.getLogger()
 
     logHandler = logging.StreamHandler()
-    formatter = jsonlogger.JsonFormatter()
+    formatter = JsonFormatter()
     logHandler.setFormatter(formatter)
     logger.addHandler(logHandler)
 ```
 
-### Customizing fields
-
-The fmt parser can also be overidden if you want to have required fields that differ from the default of just `message`.
+### Output fields
 
-These two invocations are equivalent:
+You can control the logged fields by setting the `fmt` argument when creating the formatter. By default formatters will follow the same `style` of `fmt` as the `logging` module: `%`, `$`, and `{`. All [`LogRecord` attributes](https://docs.python.org/3/library/logging.html#logrecord-attributes) can be output using their name.
 
 ```python
-class CustomJsonFormatter(jsonlogger.JsonFormatter):
-    def parse(self):
-        return self._fmt.split(';')
-
-formatter = CustomJsonFormatter('one;two')
-
-# is equivalent to:
-
-formatter = jsonlogger.JsonFormatter('%(one)s %(two)s')
+formatter = JsonFormatter("{message}{asctime}{exc_info}", style="{")
 ```
 
 You can also add extra fields to your json output by specifying a dict in place of message, as well as by specifying an `extra={}` argument.
@@ -94,9 +86,9 @@ Contents of these dictionaries will be added at the root level of the entry and
 You can also use the `add_fields` method to add to or generally normalize the set of default set of fields, it is called for every log event. For example, to unify default fields with those provided by [structlog](http://www.structlog.org/) you could do something like this:
 
 ```python
-class CustomJsonFormatter(jsonlogger.JsonFormatter):
+class CustomJsonFormatter(JsonFormatter):
     def add_fields(self, log_record, record, message_dict):
-        super(CustomJsonFormatter, self).add_fields(log_record, record, message_dict)
+        super().add_fields(log_record, record, message_dict)
         if not log_record.get('timestamp'):
             # this doesn't use record.created, so it is slightly off
             now = datetime.utcnow().strftime('%Y-%m-%dT%H:%M:%S.%fZ')
@@ -105,32 +97,55 @@ class CustomJsonFormatter(jsonlogger.JsonFormatter):
             log_record['level'] = log_record['level'].upper()
         else:
             log_record['level'] = record.levelname
+        return
 
 formatter = CustomJsonFormatter('%(timestamp)s %(level)s %(name)s %(message)s')
 ```
 
 Items added to the log record will be included in *every* log message, no matter what the format requires.
 
-### Adding custom object serialization
+You can also override the `process_log_record` method to modify fields before they are serialized to JSON.
+
+```python
+class SillyFormatter(JsonFormatter):
+    def process_log_record(log_record):
+        new_record = {k[::-1]: v for k, v in log_record.items()}
+        return new_record
+```
+
+#### Supporting custom styles
+
+It is possible to support custom `style`s by setting `validate=False` and overriding the `parse` method.
+
+For example:
+
+```python
+class CommaSupport(JsonFormatter):
+    def parse(self) -> list[str]:
+        if isinstance(self._style, str) and self._style == ",":
+            return self._fmt.split(",")
+        return super().parse()
+
+formatter = CommaSupport("message,asctime", style=",", validate=False)
+```
+
+### Custom object serialization
+
+Most formatters support `json_default` which is used to control how objects are serialized.
 
 For custom handling of object serialization you can specify default json object translator or provide a custom encoder
 
 ```python
-def json_translate(obj):
+def my_default(obj):
     if isinstance(obj, MyClass):
         return {"special": obj.special}
 
-formatter = jsonlogger.JsonFormatter(json_default=json_translate,
-                                     json_encoder=json.JSONEncoder)
-logHandler.setFormatter(formatter)
-
-logger.info({"special": "value", "run": 12})
-logger.info("classic message", extra={"special": "value", "run": 12})
+formatter = JsonFormatter(json_default=my_default)
 ```
 
 ### Using a Config File
 
-To use the module with a config file using the [`fileConfig` function](https://docs.python.org/3/library/logging.config.html#logging.config.fileConfig), use the class `pythonjsonlogger.jsonlogger.JsonFormatter`. Here is a sample config file.
+To use the module with a config file using the [`fileConfig` function](https://docs.python.org/3/library/logging.config.html#logging.config.fileConfig), use the class `pythonjsonlogger.json.JsonFormatter`. Here is a sample config file.
 
 ```ini
 [loggers]
@@ -161,6 +176,13 @@ format = %(message)s
 class = pythonjsonlogger.jsonlogger.JsonFormatter
 ```
 
+### Alternate JSON Encoders
+
+The following JSON encoders are also supported:
+
+- [orjson](https://github.com/ijl/orjson) - `pythonjsonlogger.orjon.OrjsonFormatter`
+- [msgspec](https://github.com/jcrist/msgspec) - `pythonjsonlogger.msgspec.MsgspecFormatter`
+
 ## Example Output
 
 Sample JSON with a full formatter (basically the log message from the unit test). Every log message will appear on 1 line like a typical logger.
@@ -180,7 +202,7 @@ Sample JSON with a full formatter (basically the log message from the unit test)
     "msecs": 506.24799728393555,
     "pathname": "tests/tests.py",
     "lineno": 60,
-    "asctime": ["12-05-05 22:11:08,506248"],
+    "asctime": "12-05-05 22:11:08,506248",
     "message": "testing logging format",
     "filename": "tests.py",
     "levelname": "INFO",

pylintrc

@@ -3,7 +3,7 @@
 # A comma-separated list of package or module names from where C extensions may
 # be loaded. Extensions are loading into the active Python interpreter and may
 # run arbitrary code.
-extension-pkg-whitelist=
+extension-pkg-whitelist=orjson
 
 # Add files or directories to the blacklist. They should be base names, not
 # paths.

pyproject.toml

@@ -4,18 +4,21 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "python-json-logger"
-version = "3.0.1"
+version = "3.1.0.rc1"
 description = "JSON Log Formatter for the Python Logging Package"
 authors = [
     {name = "Zakaria Zajac", email = "zak@madzak.com"},
+    {name = "Nicholas Hairs", email = "info+python-json-logger@nicholashairs.com"},
 ]
 maintainers  = [
     {name = "Nicholas Hairs", email = "info+python-json-logger@nicholashairs.com"},
 ]
 
 # Dependency Information
-requires-python = ">=3.7"
-# dependencies = []
+requires-python = ">=3.8"
+dependencies = [
+    "typing_extensions",
+]
 
 # Extra information
 readme = "README.md"
@@ -26,7 +29,6 @@ classifiers = [
     "License :: OSI Approved :: BSD License",
     "Operating System :: OS Independent",
     "Programming Language :: Python :: 3 :: Only",
-    "Programming Language :: Python :: 3.7",
     "Programming Language :: Python :: 3.8",
     "Programming Language :: Python :: 3.9",
     "Programming Language :: Python :: 3.10",
@@ -41,15 +43,18 @@ classifiers = [
 GitHub = "https://github.com/nhairs/python-json-logger"
 
 [project.optional-dependencies]
-lint = [
+dev = [
+    ## Optional but required for dev
+    "orjson;implementation_name!='pypy' and python_version<'3.13'",
+    "msgspec;implementation_name!='pypy' and python_version<'3.13'",
+    ## Lint
     "validate-pyproject[all]",
     "black",
     "pylint",
     "mypy",
-]
-
-test = [
+    ## Test
     "pytest",
+    "freezegun",
 ]
 
 [tool.setuptools.packages.find]

src/pythonjsonlogger/__init__.py

@@ -0,0 +1,40 @@
+### IMPORTS
+### ============================================================================
+## Future
+
+## Standard Library
+import warnings
+
+## Installed
+
+## Application
+import pythonjsonlogger.json
+
+### CONSTANTS
+### ============================================================================
+try:
+    import orjson
+
+    ORJSON_AVAILABLE = True
+except ImportError:
+    ORJSON_AVAILABLE = False
+
+
+try:
+    import msgspec
+
+    MSGSPEC_AVAILABLE = True
+except ImportError:
+    MSGSPEC_AVAILABLE = False
+
+
+### DEPRECATED COMPATIBILITY
+### ============================================================================
+def __getattr__(name: str):
+    if name == "jsonlogger":
+        warnings.warn(
+            "pythonjsonlogger.jsonlogger has been moved to pythonjsonlogger.json",
+            DeprecationWarning,
+        )
+        return pythonjsonlogger.json
+    raise AttributeError(f"module {__name__} has no attribute {name}")