bpo-34003: Use dict instead of OrderedDict in csv.DictReader

[selik]

DictReader can now use basic dicts instead of OrderedDict, as of version
3.7's change to specify that dict maintains keys in insertion order.
This will be more efficient and more pleasant to read at the interactive
prompt.

I also changed a list comprehension to a generator expression inside of
a ", ".join() to avoid the unnecessary list construction.

https://bugs.python.org/issue34003

[methane]

I also changed a list comprehension to a generator expression inside of
a ", ".join() to avoid the unnecessary list construction.

FYI, str.join() creates temporary list if input is not list or tuple. You can't avoid list construction.
Changing list comprehension to genexp make it little slower, because of generator overhead.

$ python3 -m timeit '",".join(str(x) for x in range(10))'
100000 loops, best of 5: 3.46 usec per loop

$ python3 -m timeit '",".join([str(x) for x in range(10)])'
100000 loops, best of 5: 3.09 usec per loop

[selik]

Doh. I should have checked the timings before I assumed. I'll revert that.

[BoboTiG]

Thanks for the patch :)

[merwok]

This entry should have been added without deleting the previous one

[GPHemsley]

Looks like this never got fixed...

[merwok]

PR #20657 fixes this

[nielskrijger]

It's a bit dissapointing that the return type of a function can suddenly change going from 3.7 to 3.8. I'll spare the details, but my assumptions about Python versioning were clearly wrong if this is allowed.

[merwok]

@nielskrijger the interface is 99% the same, and this is documented in the What’s New.

Major updates avoid gratuitous breakage, but have enhancements and fixes that require developers to test before updating.

[nielskrijger]

I assume with Major you mean a Minor (3.7 => 3.8 is not a major version).

The return value changing from OrderedDict to a normal dict is a loss of functionality; a dict and OrderedDict tend to have different characteristics https://stackoverflow.com/questions/50872498/will-ordereddict-become-redundant-in-python-3-7 , any code depending on those characteristics would break. I'm not sure how you calculcated this 99%, but if that means knowingly 1% of apps break because of this change I would have at least pointed it out as a breaking change.

Don't get me wrong, the change in itself makes perfect sense and is for the better. But I would definitely have marked this out as a breaking change.

[selik]

@nielskrijger The mailing list is probably a better place to discuss the policy. I appreciate that you changed your argument from focusing on a type change to a change in functionality. Not everyone would agree, but much of Python's design cares more about interface than type (as evidenced by "magic" methods).

Calling it a breaking change, are you saying there should have been a deprecation cycle, or something else?

If you have a specific example of someone using OrderedDict.move_to_end in this context, I'd like to see it.

[merwok]

No, I meant 3.7 to 3.8. It is a major update, or a feature release if you prefer.
3.8.4 to 3.8.5 is a minor update that takes little effort to validate, but changing your runtime to 3.8 or to 3.9 requires more effort.

(We use major in the regular sense of important; for CPython it’s not very useful to refer to X.Y.Z as major.minor.micro, it’s more line (of development) . major (or feature) . micro (or patch))

[nielskrijger]

If you have a specific example of someone using OrderedDict.move_to_end in this context, I'd like to see it.

I didn't find this out by accident. If an app is primarily about uploading, parsing and generating csvs with fairly advanced customization, I hope it's clear something might break.

Calling it a breaking change, are you saying there should have been a deprecation cycle, or something else?

I could imagine a number of solutions, and in the mailing lists I do see some evidence of these being pointed out:

• https://bugs.python.org/msg320748
• https://mail.python.org/pipermail/python-dev/2017-December/151351.html
• https://mail.python.org/pipermail/python-dev/2017-December/151360.html

One of those suggestions was a configuration option; that would have been a quick fix on my end.

But I do feel silly, as this was obviously an argument that was indeed discussed and the risk considered neglible I assume.

No, I meant 3.7 to 3.8. It is a major update, [...] (We use major in the regular sense of important; for CPython it’s not very useful to refer to X.Y.Z as major.minor.micro, it’s more line (of development) . major (or feature) . micro (or patch))

Sorry, I didn't know that. I think I've got semver a bit too stuck in my mind I guess, you certainly helped me to point this out in Python's versioning. Thanks!

[methane]

Note that we changed the returned type from dict to OrderedDict in Python 3.6 without deprecation period. It clearly breaks == behavior so it might be breaking change for very minor use cases.

We regularly change behavior which can be breaking change some people when:

• Using deprecation period is not simple.
• Estimated affected users is very small.

Ideally speaking, we should add OrderedDictReader instead of changing DictReader in Python 3.6.
But it is too late. Unnecessary breaking change already happened.

[merwok]

It clearly breaks == behavior

It doesn’t, comparing an ordered dict to a dict works (looks at the items as a set)

[methane]

It doesn’t, comparing an ordered dict to a dict works (looks at the items as a set)

Consider comparing two csv files. When column order is changed but data is not changed, row1 == row2 in Python 3.5 and Python 3.8+, but row1 != row2 in Python 3.6, and Python 3.7.

It doesn't emit any error so user may not notice their scripts is broken in Python 3.6. It might be worse than missing .move_to_end() in Python 3.8, because user can notice the missing .move_to_end() when they are affected.

[merwok]

Sorry, I don’t follow what you are saying, and this is the wrong place for the discussion!