Metadata-Version: 2.1
Name: gpt3-tokenizer
Version: 0.1.0
Summary: Encoder/Decoder and tokens counter for GPT3
Home-page: https://github.com/alisonjf/gpt3-tokenizer
License: MIT
Keywords: openai,gpt,gpt-3,gpt3,gpt4,gpt-4,tokenizer
Author: Alison Ferrenha
Requires-Python: >=2.7
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.4
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Requires-Dist: regex (==2021.11.10)
Project-URL: Repository, https://github.com/alisonjf/gpt3-tokenizer
Description-Content-Type: text/x-rst

gpt3_tokenizer
===============
| An `OpenAI`_ GPT3 helper library for encoding/decoding strings and counting tokens.
| Counting tokens gives the same output as OpenAI's `tokenizer`_
| Works with any Python version starting from `2.7.18`_

Installing
--------------
.. code-block:: bash

    pip install gpt3_tokenizer

    
Examples:
---------------------

**Encoding/decoding a string**

.. code-block:: python

    import gpt3_tokenizer

    a_string = "That's my beautiful and sweet string"
    encoded = gpt3_tokenizer.encode(a_string) # outputs [2504, 338, 616, 4950, 290, 6029, 4731]
    decoded = gpt3_tokenizer.decode(encoded) # outputs "That's my beautiful and sweet string"

**Counting tokens**

.. code-block:: python

    import gpt3_tokenizer

    a_string = "That's my beautiful and sweet string"
    tokens_count = gpt3_tokenizer.count_tokens(a_string) # outputs 7

.. _tokenizer: https://platform.openai.com/tokenizer
.. _OpenAI: https://openai.com/
.. _2.7.18: https://www.python.org/downloads/release/python-2718/
