Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31408

Build Spark’s own datetime pattern definition

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • SQL
    • None

    Description

      This is an umbrella ticket for building Spark's own Datetime patterns and related works.

      In Spark version 2.4 and earlier, datetime parsing and formatting are performed by the old Java 7 `SimpleDateFormat` API. Since Spark 3.0, we switch to the new Java 8 `DateTimeFormatter` to use the Proleptic Gregorian calendar, which is required by the ISO and SQL standards.

      However, there are some datetime patterns not compatible between Java 8 and Java 7 APIs, and it's fragile to rely on the JDK API to define Spark's behavior. We should build our own Datetime patterns, which is compatible with Spark 2.4 (the old Java 7 `SimpleDateFormat` API).

      Attachments

        1.
        		 Disable week-based date filed for parsing Sub-task Closed Kent Yao 2
        2.
        		 Backward Compatibility for Parsing and Formatting Datetime Sub-task Closed Yuanjian Li
        3.
        		 Parsing seconds fraction with variable length for timestamp Sub-task Closed Kent Yao 2
        4.
        		 Remove the unnecessary config spark.sql.legacy.timeParser.enabled Sub-task Closed Kent Yao 2
        5.
        		 Fix errors and missing parts for datetime pattern document Sub-task Closed Kent Yao 2
        6.
        		 Remove support for 'e'/'c' as datetime pattern charactar Sub-task Closed Kent Yao 2
        7.
        		 Raise exception instead of silent change for new DateFormatter Sub-task Closed Yuanjian Li
        8.
        		 Performance regression with new TimestampFormatter for json and csv Sub-task Closed Kent Yao 2
        9.
        		 allow missing year/hour when parsing date/timestamp Sub-task Closed Wenchen Fan
        10.
        		 Fix perf regression of date/timestamp formatting in toHiveString Sub-task Closed Max Gekk
        11.
        		 Format Java date-time types in Row.jsonValue directly Sub-task Closed Max Gekk
        12.
        		 Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q' Sub-task Closed Kent Yao 2
        13.
        		 Fail datetime parsing/formatting if detect the Java 8 bug of stand-alone form Sub-task Closed Wenchen Fan
        14.
        		 Consistent error handling for datetime formatting functions Sub-task Closed Kent Yao 2
        15.
        		 Fix silent data change for datetime formatting Sub-task Closed Kent Yao 2
        16.
        		 First day of week changed for non-MONDAY_START Lacales Sub-task Closed Kent Yao 2
        17.
        		 Handle am-pm timestamp parsing when hour is missing Sub-task Closed Kent Yao 2
        18.
        		 Forbid datetime pattern letter u Sub-task Closed Wenchen Fan
        19.
        		 Fix Parsing day of year when year field pattern is missing Sub-task Closed Kent Yao 2
        20.
        		 Adjacent value parsing not supported for Localized Patterns because of JDK bug Sub-task Resolved Unassigned
        21.
        		 Fix silent data change for timestamp parsing if overflow happens Sub-task Closed Kent Yao 2

        Activity

          People

            Unassigned Unassigned
            XuanYuan Yuanjian Li
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: