spark structured dataset streaming
-
Structured Streaming - Datasetsspark streaming 2022. 4. 7. 21:24
문제 Dataset 을 Streaming 에 활용해 보자 코드 import spark.implicits._ val carsDS: Dataset[Car] = spark.readStream .format("socket") .option("host", "localhost") .option("port", 12345) .load() // DF with single string column "value" .select(from_json(col("value"), carsSchema).as("car")) // composite column (struct) .selectExpr("car.*") // DF with multiple columns .as[Car] // encoder can be passed implicitl..