Sqlcmd could support some codepage options

[shueybubbles]

We haven't yet implemented this switch:

-f codepage | i:codepage[,o:codepage] | o:codepage[,i:codepage]

The charmap package has a bunch of Windows code page decoders and encoders we could leverage to implement this switch.

[shueybubbles]

go-sqlcmd also does not convert files from the system code page; it assumes a file with no BOM is UTF8

[emwl]

We just ran into this issue as well. One of our tools runs a number of SQL files by calling sqlcmd.exe and passing the .sql file plus various CLI options. We do this mostly because we don't really have any useful way of running DDL statements in bulk from .NET Code without having to parse the SQL file and hoping we can find useful locations that we can send as individual SqlCommand or similar without breaking.

One of them is -f 65001 as our files saved by SSMS default to "Unicode (UTF-8 with signature)" which caused us issues before. But because we ended up on a system that has go-sqlcmd rather than the old ODBC-based one, it simply failed without giving us any useful output. We only figured this out afterwards by manually running our command just to see that -f was not supported (despite being documented!)

It would be nice if the option could simply be ignored (either for now, or simply forever as it seems to handle the BOM just fine without an option) as those CLI options are there for backwards compatibility after all.

[shueybubbles]

I think we could support just 65001 until such time we can invest in real code page support. Silently ignoring other code page values could lead to data corruption.