代码:
public class SparkToDorisConstants {
public static final String DORIS_DBNAME = "test_db";
public static final String DORIS_TABLE_NAME = "example_list_tbl";
public static final String DORIS_FE_NODES = "192.168.18.194:31620";
public static final String DORIS_USER = "root";
public static final String DORIS_PASSWORD = "PleaseChangeMe";
public static final String HDFS_PATH = "";
public static final String CHECKPOINT_DIRECTORY = "";
public static final String KAFKA_BOOTSTRAP_SERVERS = "";
public static final String KAFKA_TOPIC_NAME = "";
public static final String KAFKA_OFFSET_MODEL = "earliest";
}
public static void main( String[] args )
{
//创建sparkconf
SparkConf sparkConf = new SparkConf()
.setAppName("SparkDorisConnectorDemo")
.setMaster("local[*]");
SparkSession sparkSession = SparkSession.builder()
.config(sparkConf)
.getOrCreate();
Encoder<Example_list_tbl> rowEncoder = Encoders.bean(Example_list_tbl.class);
Dataset<Example_list_tbl> dorisSparkDF = sparkSession
.read()
.format("doris")
.option("doris.table.identifier", SparkToDorisConstants.DORIS_DBNAME.concat(".").concat(SparkToDorisConstants.DORIS_TABLE_NAME))
.option("doris.fenodes", SparkToDorisConstants.DORIS_FE_NODES)
.option("user", SparkToDorisConstants.DORIS_USER)
.option("password", SparkToDorisConstants.DORIS_PASSWORD)
.load()
.as(rowEncoder);
dorisSparkDF.show(5);
sparkSession.close();
}
错误信息:
utput: user_id#10, date#11, timestamp#12, city#13, age#14, sex#15, last_visit_date#16, cost#17L, max_dwell_time#18, min_dwell_time#19,能看到表的信息,说明连接设置应当是正确的,可是还是报错了!下面是报错信,请高手给个解答,谢!
25/08/13 17:22:33 WARN DorisBackendThriftClient: Connect to doris Backend{host='doris-cluster-be-0.doris-cluster-be-internal.doris.svc.cluster.local', httpPort=-1, rpcPort=9060} failed.
org.apache.doris.shaded.org.apache.thrift.transport.TTransportException: java.net.UnknownHostException: doris-cluster-be-0.doris-cluster-be-internal.doris.svc.cluster.local
at org.apache.doris.shaded.org.apache.thrift.transport.TSocket.open(TSocket.java:255)
at org.apache.doris.spark.client.DorisBackendThriftClient.open(DorisBackendThriftClient.java:89)
at org.apache.doris.spark.client.DorisBackendThriftClient.(DorisBackendThriftClient.java:67)
at org.apache.doris.spark.client.read.AbstractThriftReader.(AbstractThriftReader.java:82)
at org.apache.doris.spark.client.read.DorisThriftReader.(DorisThriftReader.java:28)
at org.apache.doris.spark.read.DorisPartitionReader.reader$lzycompute(DorisPartitionReader.scala:42)
at org.apache.doris.spark.read.DorisPartitionReader.reader(DorisPartitionReader.scala:40)
at org.apache.doris.spark.read.DorisPartitionReader.next(DorisPartitionReader.scala:50)
at org.apache.spark.sql.execution.datasources.v2.PartitionIterator.hasNext(DataSourceRDD.scala:93)