Skip to content

Document PyUnicode_* API #46236

Closed
Closed
Document PyUnicode_* API#46236
Labels
docsDocumentation in the Doc dirtopic-unicodetype-featureA feature request or enhancement

Description

@avassalotti
avassalotti
opened on Jan 27, 2008
BPO 1944
Nosy @malemburg, @birkenfeld, @vstinner, @avassalotti, @berkerpeksag, @vadmium, @serhiy-storchaka, @shihai1991, @furkanonder
PRs
  • bpo-1944: wrap functions with macro #20011
    		•
    Files
  • unicode.patch: docs for PyUnicodes C-API functions: FromFormat, FromFormatV, FromString, FromStringAndSize, Partition, RPartition and RSplit
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2008-01-27.06:26:42.019>
labels = ['type-feature', 'docs']
title = 'Document PyUnicode_* API'
updated_at = <Date 2020-06-21.12:39:22.129>
user = 'https://github.com/avassalotti'

    bugs.python.org fields:

    activity = <Date 2020-06-21.12:39:22.129>
actor = 'shihai1991'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2008-01-27.06:26:42.019>
creator = 'alexandre.vassalotti'
dependencies = []
files = ['13717']
hgrepos = []
issue_num = 1944
keywords = ['patch']
message_count = 9.0
messages = ['61734', '86116', '86121', '89100', '185552', '185725', '264571', '264575', '368582']
nosy_count = 11.0
nosy_names = ['lemburg', 'georg.brandl', 'vstinner', 'alexandre.vassalotti', 'donlorenzo', 'docs@python', 'berker.peksag', 'martin.panter', 'serhiy.storchaka', 'shihai1991', 'furkanonder']
pr_nums = ['20011']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue1944'
versions = ['Python 3.5', 'Python 3.6']

    [edit @encukou]: Converting the lists below to checkboxes:

    • PyUnicode_Resize
    • PyUnicode_InternImmortal (removed in 3.12)
    • PyUnicode_GetDefaultEncoding
    • PyUnicode_SetDefaultEncoding (removed in 3.2)
    • PyUnicode_BuildEncodingMap
    • PyUnicode_FromFormatV
    • PyUnicode_UTF7
    • PyUnicode_AsEncodedObject
    • PyUnicode_FromOrdinal
    • PyUnicode_DecodeFSDefault
    • PyUnicode_DecodeFSDefaultAndSize
    • PyUnicode_DecodeUTF7
    • PyUnicode_DecodeUTF7Stateful
    • PyUnicode_EncodeDecimal
    • PyUnicode_EncodeUTF7
    • PyUnicode_FromFormat
    • PyUnicode_FromString
    • PyUnicode_FromStringAndSize
    • PyUnicode_GetMax
    • PyUnicode_Partition
    • PyUnicode_RPartition
    • PyUnicode_RSplit
    • PyUnicode_IsIdentifier
    • PyUnicode_Append
    • PyUnicode_AppendAndDel
    • Py_UNICODE_REPLACEMENT_CHARACTER
    • PyUnicodeIter_Type
    • PyUnicode_AsDecodedObject
    • PyUnicode_AsDecodedUnicode
    • PyUnicode_AsEncodedUnicode
    • _PyUnicode_ClearStaticStrings (removed in 3.9)
    • _PyUnicode_EQ (removed in 3.14)
    • _PyUnicode_FromId (private)

    Linked PRs

    Activity

    1. avassalotti commented on Jan 27, 2008

      @avassalotti
      avassalotti
      on Jan 27, 2008
      MemberAuthor

      I was wandering whether the pointer returned by PyUnicode_AsString needs
      to be freed after usage (It turned it doesn't since the result is
      cached). However, I found out that there isn't any documentation on
      docs.python.org about the PyUnicode_AsString and
      PyUnicode_AsStringAndSize functions. Although, both are documented in
      the public unicodeobject.h header.

      I notice that the documentation for several other unicode functions is
      missing. Quickly, I see:

      PyUnicode_Resize
      PyUnicode_InternImmortal
      PyUnicode_GetDefaultEncoding
      PyUnicode_SetDefaultEncoding
      PyUnicode_BuildEncodingMap
      PyUnicode_FromFormatV
      PyUnicode_*UTF7*
      PyUnicode_AsEncodedObject
      PyUnicode_FromOrdinal
      PyUnicode_DecodeFSDefault
      PyUnicode_DecodeFSDefaultAndSize

      It would probably be a good idea to polish up the documentation for
      PyUnicode as much as possible for Python 3000, since extension
      developers will certainly need to refer to it a lot during the
      transition from 2.x.

    2. avassalotti
      added
      easy
      docsDocumentation in the Doc dir
      on Jan 27, 2008
    3. tiran
      added
      type-featureA feature request or enhancement
      on Jan 27, 2008

    4. donlorenzo commented on Apr 18, 2009

      @donlorenzo
      donlorenzomannequin
      on Apr 18, 2009
      Mannequin

      In addition to the above mentioned functions I found these to be
      undocumented:

      PyUnicode_DecodeUTF7
      PyUnicode_DecodeUTF7Stateful
      PyUnicode_EncodeDecimal
      PyUnicode_EncodeUTF7
      PyUnicode_FromFormat
      PyUnicode_FromString
      PyUnicode_FromStringAndSize
      PyUnicode_GetMax
      PyUnicode_Partition
      PyUnicode_RPartition
      PyUnicode_RSplit

      From the original list the following functions seem to have been removed:

      PyUnicode_InternImmortal
      PyUnicode_DecodeFSDefault
      PyUnicode_DecodeFSDefaultAndSize

      I try to put together a patch for some of these during the weekend.

    5. donlorenzo commented on Apr 18, 2009

      @donlorenzo
      donlorenzomannequin
      on Apr 18, 2009
      Mannequin

      Ok, here is my shot at a patch for at least some of the undocumented
      functions. Namely the following functions are being documented in the patch:

      PyUnicode_FromFormat
      PyUnicode_FromFormatV
      PyUnicode_FromString
      PyUnicode_FromStringAndSize
      PyUnicode_Partition
      PyUnicode_RPartition
      PyUnicode_RSplit

      Please thoroughly review this patch since I didn't really digg into the
      source to find out what the functions do but rather just copied old
      PyString documentation or derived it from the docs for the Python API.

    6. avassalotti commented on Jun 8, 2009

      @avassalotti
      avassalotti
      on Jun 8, 2009
      MemberAuthor

      The patch looks alright. I don't like the documentation for
      PyUnicode_FromFormatV, however. Here's my attempt to document it:

      .. cfunction:: PyObject* PyUnicode_FromFormatV(const char *format,
      va_list vargs)

      Equivalent to the function :cfunc:`PyUnicode_FromFormat`, except that
      it takes a va_list instead of variable number of arguments.

    7. birkenfeld
      self-assigned this
      on Apr 3, 2010
    8. BreamoreBoy
      assigned
      docspython
      and unassigned
      birkenfeld
      on Sep 20, 2010

    9. BreamoreBoy commented on Mar 30, 2013

      @BreamoreBoy
      BreamoreBoymannequin
      on Mar 30, 2013
      Mannequin

      Is it worth applying the patch given the complete rewrite of unicode for 3.3 via PEP-393?

    10. malemburg commented on Apr 1, 2013

      @malemburg
      malemburg
      on Apr 1, 2013
      Member

      On 30.03.2013 13:09, Mark Lawrence wrote:

      Is it worth applying the patch given the complete rewrite of unicode for 3.3 via PEP-393?

      PEP-393 only changed the way Unicode is internally stored.
      The Unicode API is mostly unaffected by this change.

    11. berkerpeksag commented on Apr 30, 2016

      @berkerpeksag
      berkerpeksag
      on Apr 30, 2016
      Member

      Remaining undocumented functions:

      From this issue:

      PyUnicode_RSplit
      PyUnicode_Partition
      PyUnicode_RPartition

      From bpo-10435:

      PyUnicode_IsIdentifier
      PyUnicode_Append
      PyUnicode_AppendAndDel
      PyUnicode_GetDefaultEncoding
      PyUnicode_FromOrdinal
      PyUnicode_Resize
      PyUnicode_GetMax
      PyUnicode_InternImmortal
      PyUnicode_CHECK_INTERNED

      From bpo-18688:

      Py_UNICODE_REPLACEMENT_CHARACTER
      PyUnicodeIter_Type
      PyUnicode_AsDecodedObject
      PyUnicode_AsDecodedUnicode
      PyUnicode_AsEncodedObject
      PyUnicode_AsEncodedUnicode
      PyUnicode_BuildEncodingMap

    12. berkerpeksag
      removed
      easy
      on Apr 30, 2016
    13. berkerpeksag
      changed the title [-]Documentation for PyUnicode_AsString (et al.) missing.[/-] [+]Document PyUnicode_* API[/+] on Apr 30, 2016

    14. 38 remaining items

    15. vstinner commented on Apr 22, 2025

      @vstinner
      vstinner
      on Apr 22, 2025
      Member

      I agree, though they have been deprecated for many years but there is no planned removal date that I can find. Should we just remove them now?

      IMO now it's too late for the 3.14 release cycle. It would be better to schedule such removal at the start of a new dev cycle, such as Python 3.15. Someone also has to check if these functions are used in the wild (ex: run a code search).

    16. StanFromIreland commented on Apr 22, 2025

      @StanFromIreland
      StanFromIreland
      on Apr 22, 2025
      Member

      cc @serhiy-storchaka who originally deprecated them in 0093907

      A new issue is probably best for this, scheduling for 3.15 seems good for now.

    17. encukou
      added a commit that references this issue on Apr 29, 2025

      gh-46236: Document `PyUnicodeIter_Type` (GH-132925)

      bba14c3
    18. StanFromIreland
      added 2 commits that reference this issue on Apr 29, 2025

      pythongh-46236: Document `PyUnicodeIter_Type` (pythonGH-132925)

      531f197

      pythongh-46236: Document `PyUnicodeIter_Type` (pythonGH-132925)

      c331d01
    19. encukou
      added a commit that references this issue on Apr 29, 2025

      [3.13] gh-46236: Document `PyUnicodeIter_Type` (GH-132925) (GH-133149)

      ff41346

    20. StanFromIreland commented on May 1, 2025

      @StanFromIreland
      StanFromIreland
      on May 1, 2025
      Member

      The list needs to be updated, just PyUnicode_BuildEncodingMap is left.

    21. vstinner
      added a commit that references this issue on May 9, 2025

      gh-46236: Document PyUnicode_BuildEncodingMap (#133270)

      f34ec09
    22. miss-islington
      added a commit that references this issue on May 9, 2025

      pythongh-46236: Document PyUnicode_BuildEncodingMap (pythonGH-133270)

      c5e7460

    23. vstinner commented on May 9, 2025

      @vstinner
      vstinner
      on May 9, 2025
      Member

      The list is complete. I close the issue. Thanks to everybody who was involved in this old issue (2008!).

    24. vstinner
      closed this as completedon May 9, 2025
    25. vstinner
      added 2 commits that reference this issue on May 9, 2025

      [3.14] gh-46236: Document PyUnicode_BuildEncodingMap (GH-133270) (#13…

      9ad1516

      [3.13] gh-46236: Document PyUnicode_BuildEncodingMap (#133770)

      78a04b3
    26. Pranjal095
      added a commit that references this issue on Jul 12, 2025

      pythongh-46236: Document PyUnicode_BuildEncodingMap (python#133270)

      8998c09
    27. taegyunkim
      added a commit that references this issue on Aug 4, 2025

      pythongh-46236: Document PyUnicode_BuildEncodingMap (python#133270)

      29d11d2
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      docsDocumentation in the Doc dirtopic-unicodetype-featureA feature request or enhancement

      Projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions