HPE Community
HPE Ezmeral Software platform
1822147 Members
4147 Online
109640 Solutions
New Discussion юеВ

Need help with abnormal Apache Drill Behavior

 
CKSunny
Occasional Visitor

тАО09-08-2021 08:00 PM - last edited on тАО09-09-2021 08:02 PM by

тАО09-08-2021 08:00 PM - last edited on тАО09-09-2021 08:02 PM by

Need help with abnormal Apache Drill Behavior

Hi All,

I am currently running a 4 node Drill Cluster with Core 6.1.1 MEP 6.3.3 on RHEL 7.9 vms.

I uses pydrill to send query request to individual node for table creation.

For some long query (around 200k+ records), the job failed after a minute and return a Failed status with not enough heap memory error. However, when I tried to rerun the query, I find that the table is already created, but the record count is wrong.

Anyone have the experience with similar issue?

Is there any way to fix the issue without adding extra heap memory? Or is there any config I can do to prevent Apache Drill from creating incomplete/failed query tables? Or as last resort, is there way for me to capture these kind of behavior in pydrill/API level?

0 Kudos
Reply
6 REPLIES 6
Harshkohli
HPE Pro

тАО09-08-2021 08:45 PM

тАО09-08-2021 08:45 PM

Re: Need help with abnormal Apache Drill Behavior

Thanks for the post. Can you paste the complete stacktrace of the error?  Can you paste the api calls also?

Kindly refer below article as well and see if it helps?

https://support.datafabric.hpe.com/s/article/Troubleshooting-DRILL-OutOfMemory-error-and-possible-workarounds?language=en_US

I work for HPE
0 Kudos
Reply
CKSunny
Occasional Visitor

тАО09-10-2021 12:07 AM - edited тАО09-10-2021 12:19 AM

тАО09-10-2021 12:07 AM - edited тАО09-10-2021 12:19 AM

Re: Need help with abnormal Apache Drill Behavior

The query is submit to individual drillbit node port 8047 with pydrill.client module:

 

PyDrill("hostname", 8047).query(line, timeout=180)

 

Failed query as below:

create table dfs.`prd_tmp`.`AAAA_STG_Transaction` as select CAST(`Block` as VARCHAR(100)) as `Block`, CAST(`Building Name` as VARCHAR(1000)) as `Building Name`, CAST(`Building Name in Chinese` as VARCHAR(500)) as `Building Name in Chinese`, CAST(`Building Name in English` as VARCHAR(500)) as `Building Name in English`, CAST(`Building Remarks` as VARCHAR(3000)) as `Building Remarks`, CAST(`Consideration Remarks` as VARCHAR(3000)) as `Consideration Remarks`, CAST(`DD/Lot No.` as VARCHAR(500)) as `DD/Lot No.`, CAST(`Data Source` as VARCHAR(100)) as `Data Source`, CAST(`Delivery Date` as VARCHAR(100)) as `Delivery Date`, CAST(`Direction` as VARCHAR(100)) as `Direction`, CAST(`District` as VARCHAR(100)) as `District`, CAST(`District 1` as VARCHAR(100)) as `District 1`, CAST(`District 1 in Chinese` as VARCHAR(100)) as `District 1 in Chinese`, CAST(`District 1 in English` as VARCHAR(100)) as `District 1 in English`, CAST(`District 2` as VARCHAR(100)) as `District 2`, CAST(`District 2 in Chinese` as VARCHAR(100)) as `District 2 in Chinese`, CAST(`District 2 in English` as VARCHAR(100)) as `District 2 in English`, CAST(`Estate Name` as VARCHAR(1000)) as `Estate Name`, CAST(`Estate Name in Chinese` as VARCHAR(500)) as `Estate Name in Chinese`, CAST(`Estate Name in English` as VARCHAR(500)) as `Estate Name in English`, CAST(`Facilities` as VARCHAR(1000)) as `Facilities`, CAST(`Floor` as VARCHAR(100)) as `Floor`, CAST(`Floor Plan` as VARCHAR(500)) as `Floor Plan`, CAST(`Gross Area (square feet)` as VARCHAR(500)) as `Gross Area (square feet)`, CAST(`Gross Unit Price ($/square foot)` as VARCHAR(500)) as `Gross Unit Price ($/square foot)`, CAST(`Land Premium` as VARCHAR(100)) as `Land Premium`, CAST(`Lease Term` as VARCHAR(100)) as `Lease Term`, CAST(`Memo No.` as VARCHAR(100)) as `Memo No.`, CAST(`Nature` as VARCHAR(100)) as `Nature`, CAST(`Occupation Date` as VARCHAR(100)) as `Occupation Date`, CAST(`Phase` as VARCHAR(100)) as `Phase`, CAST(`Price ($)` as VARCHAR(500)) as `Price ($)`, CAST(`Property No (PRN)` as VARCHAR(1000)) as `Property No (PRN)`, CAST(`Property Type` as VARCHAR(100)) as `Property Type`, CAST(`Property Type Detail` as VARCHAR(100)) as `Property Type Detail`, CAST(`Region` as VARCHAR(100)) as `Region`, CAST(`Saleable Area (square feet)` as VARCHAR(500)) as `Saleable Area (square feet)`, CAST(`Saleable Unit Price ($/square foot)` as VARCHAR(500)) as `Saleable Unit Price ($/square foot)`, CAST(`Same Assignee & Assignor Indicator` as VARCHAR(100)) as `Same Assignee & Assignor Indicator`, CAST(`Signed/Order Date` as VARCHAR(100)) as `Signed/Order Date`, CAST(`Special Transaction` as VARCHAR(100)) as `Special Transaction`, CAST(`Street` as VARCHAR(1000)) as `Street`, CAST(`Street Name` as VARCHAR(1000)) as `Street Name`, CAST(`Street in Chinese` as VARCHAR(1000)) as `Street in Chinese`, CAST(`Street in English` as VARCHAR(1000)) as `Street in English`, CAST(`Transaction Remarks` as VARCHAR(1000)) as `Transaction Remarks`, CAST(`Unit` as VARCHAR(100)) as `Unit`, CAST(`Usage` as VARCHAR(100)) as `Usage`, CAST(`View` as VARCHAR(100)) as `View`, CAST(`System Estate ID` as VARCHAR(100)) as `System Estate ID`, CAST(`Age` as VARCHAR(500)) as `Age`, CAST(`Bay Window (square feet)` as VARCHAR(100)) as `Bay Window (square feet)`, CAST(`Efficiency Ratio` as VARCHAR(500)) as `Efficiency Ratio`, CAST(`Gross Area (square meters)` as VARCHAR(500)) as `Gross Area (square meters)`, CAST(`Residual Term` as VARCHAR(500)) as `Residual Term`, CAST(`Saleable Area (square meters)` as VARCHAR(500)) as `Saleable Area (square meters)`, CAST(`Year of Land Premium` as VARCHAR(500)) as `Year of Land Premium` from dfs.`prd_src`.`AAAA_Transaction`

 

 

The query is selecting from a folder with 15k+ json files.

Below is the drillbit.log file output during the query run time:

 

 

2021-09-09 13:16:24,510 [1ec66757-57c0-8dbb-fc0d-f4bec5e8cef9:frag:1:2] INFO  o.a.d.e.w.fragment.FragmentExecutor - 1ec66757-57c0-8dbb-fc0d-f4bec5e8cef9:1:2: State change requested AWAITING_ALLOCATION --> RUNNING
2021-09-09 13:16:24,511 [1ec66757-57c0-8dbb-fc0d-f4bec5e8cef9:frag:1:2] INFO  o.a.d.e.w.f.FragmentStatusReporter - 1ec66757-57c0-8dbb-fc0d-f4bec5e8cef9:1:2: State to report: RUNNING
2021-09-09 13:16:24,524 [1ec66757-57c0-8dbb-fc0d-f4bec5e8cef9:frag:1:2] WARN  o.a.d.e.e.ExpressionTreeMaterializer - Unable to find value vector of path `Age`, returning null instance.
2021-09-09 13:16:24,524 [1ec66757-57c0-8dbb-fc0d-f4bec5e8cef9:frag:1:2] WARN  o.a.d.e.e.ExpressionTreeMaterializer - Unable to find value vector of path `Bay Window (square feet)`, returning null instance.
2021-09-09 13:16:24,524 [1ec66757-57c0-8dbb-fc0d-f4bec5e8cef9:frag:1:2] WARN  o.a.d.e.e.ExpressionTreeMaterializer - Unable to find value vector of path `Efficiency Ratio`, returning null instance.
2021-09-09 13:16:24,524 [1ec66757-57c0-8dbb-fc0d-f4bec5e8cef9:frag:1:2] WARN  o.a.d.e.e.ExpressionTreeMaterializer - Unable to find value vector of path `Gross Area (square meters)`, returning null instance.
2021-09-09 13:16:24,524 [1ec66757-57c0-8dbb-fc0d-f4bec5e8cef9:frag:1:2] WARN  o.a.d.e.e.ExpressionTreeMaterializer - Unable to find value vector of path `Residual Term`, returning null instance.
2021-09-09 13:16:24,524 [1ec66757-57c0-8dbb-fc0d-f4bec5e8cef9:frag:1:2] WARN  o.a.d.e.e.ExpressionTreeMaterializer - Unable to find value vector of path `Saleable Area (square meters)`, returning null instance.
2021-09-09 13:16:24,524 [1ec66757-57c0-8dbb-fc0d-f4bec5e8cef9:frag:1:2] WARN  o.a.d.e.e.ExpressionTreeMaterializer - Unable to find value vector of path `Year of Land Premium`, returning null instance.
2021-09-09 13:16:24,552 [1ec66757-57c0-8dbb-fc0d-f4bec5e8cef9:frag:1:2] WARN  o.a.d.e.e.ExpressionTreeMaterializer - Unable to find value vector of path `Bay Window (square feet)`, returning null instance.
2021-09-09 13:16:24,553 [1ec66757-57c0-8dbb-fc0d-f4bec5e8cef9:frag:1:2] WARN  o.a.d.e.e.ExpressionTreeMaterializer - Unable to find value vector of path `Efficiency Ratio`, returning null instance.
2021-09-09 13:16:24,553 [1ec66757-57c0-8dbb-fc0d-f4bec5e8cef9:frag:1:2] WARN  o.a.d.e.e.ExpressionTreeMaterializer - Unable to find value vector of path `Gross Area (square meters)`, returning null instance.
2021-09-09 13:16:24,553 [1ec66757-57c0-8dbb-fc0d-f4bec5e8cef9:frag:1:2] WARN  o.a.d.e.e.ExpressionTreeMaterializer - Unable to find value vector of path `Residual Term`, returning null instance.
2021-09-09 13:16:24,553 [1ec66757-57c0-8dbb-fc0d-f4bec5e8cef9:frag:1:2] WARN  o.a.d.e.e.ExpressionTreeMaterializer - Unable to find value vector of path `Saleable Area (square meters)`, returning null instance.
2021-09-09 13:16:24,553 [1ec66757-57c0-8dbb-fc0d-f4bec5e8cef9:frag:1:2] WARN  o.a.d.e.e.ExpressionTreeMaterializer - Unable to find value vector of path `Year of Land Premium`, returning null instance.
2021-09-09 13:16:24,558 [1ec66757-57c0-8dbb-fc0d-f4bec5e8cef9:frag:1:2] INFO  o.a.d.e.c.ClassCompilerSelector - Java compiler policy: DEFAULT, Debug option: true
2021-09-09 13:16:27,064 [BitServer-2] WARN  o.a.d.exec.rpc.control.WorkEventBus - Fragment 1ec66757-57c0-8dbb-fc0d-f4bec5e8cef9:1:2 manager is not found in the work bus.
2021-09-09 13:16:27,065 [BitServer-2] INFO  o.a.d.e.w.fragment.FragmentExecutor - 1ec66757-57c0-8dbb-fc0d-f4bec5e8cef9:1:2: State change requested RUNNING --> CANCELLATION_REQUESTED
2021-09-09 13:16:27,065 [BitServer-2] INFO  o.a.d.e.w.f.FragmentStatusReporter - 1ec66757-57c0-8dbb-fc0d-f4bec5e8cef9:1:2: State to report: CANCELLATION_REQUESTED
2021-09-09 13:16:32,112 [1ec6674f-4541-f8a7-65ee-4a3d5cf63210:foreman] INFO  o.a.drill.exec.work.foreman.Foreman - Query text for query id 1ec6674f-4541-f8a7-65ee-4a3d5cf63210: create table dfs.`prd_tmp`.`AAAA_STG_Transaction` as select CAST(`Block` as VARCHAR(100)) as `Block`, CAST(`Building Name` as VARCHAR(1000)) as `Building Name`, CAST(`Building Name in Chinese` as VARCHAR(500)) as `Building Name in Chinese`, CAST(`Building Name in English` as VARCHAR(500)) as `Building Name in English`, CAST(`Building Remarks` as VARCHAR(3000)) as `Building Remarks`, CAST(`Consideration Remarks` as VARCHAR(3000)) as `Consideration Remarks`, CAST(`DD/Lot No.` as VARCHAR(500)) as `DD/Lot No.`, CAST(`Data Source` as VARCHAR(100)) as `Data Source`, CAST(`Delivery Date` as VARCHAR(100)) as `Delivery Date`, CAST(`Direction` as VARCHAR(100)) as `Direction`, CAST(`District` as VARCHAR(100)) as `District`, CAST(`District 1` as VARCHAR(100)) as `District 1`, CAST(`District 1 in Chinese` as VARCHAR(100)) as `District 1 in Chinese`, CAST(`District 1 in English` as VARCHAR(100)) as `District 1 in English`, CAST(`District 2` as VARCHAR(100)) as `District 2`, CAST(`District 2 in Chinese` as VARCHAR(100)) as `District 2 in Chinese`, CAST(`District 2 in English` as VARCHAR(100)) as `District 2 in English`, CAST(`Estate Name` as VARCHAR(1000)) as `Estate Name`, CAST(`Estate Name in Chinese` as VARCHAR(500)) as `Estate Name in Chinese`, CAST(`Estate Name in English` as VARCHAR(500)) as `Estate Name in English`, CAST(`Facilities` as VARCHAR(1000)) as `Facilities`, CAST(`Floor` as VARCHAR(100)) as `Floor`, CAST(`Floor Plan` as VARCHAR(500)) as `Floor Plan`, CAST(`Gross Area (square feet)` as VARCHAR(500)) as `Gross Area (square feet)`, CAST(`Gross Unit Price ($/square foot)` as VARCHAR(500)) as `Gross Unit Price ($/square foot)`, CAST(`Land Premium` as VARCHAR(100)) as `Land Premium`, CAST(`Lease Term` as VARCHAR(100)) as `Lease Term`, CAST(`Memo No.` as VARCHAR(100)) as `Memo No.`, CAST(`Nature` as VARCHAR(100)) as `Nature`, CAST(`Occupation Date` as VARCHAR(100)) as `Occupation Date`, CAST(`Phase` as VARCHAR(100)) as `Phase`, CAST(`Price ($)` as VARCHAR(500)) as `Price ($)`, CAST(`Property No (PRN)` as VARCHAR(1000)) as `Property No (PRN)`, CAST(`Property Type` as VARCHAR(100)) as `Property Type`, CAST(`Property Type Detail` as VARCHAR(100)) as `Property Type Detail`, CAST(`Region` as VARCHAR(100)) as `Region`, CAST(`Saleable Area (square feet)` as VARCHAR(500)) as `Saleable Area (square feet)`, CAST(`Saleable Unit Price ($/square foot)` as VARCHAR(500)) as `Saleable Unit Price ($/square foot)`, CAST(`Same Assignee & Assignor Indicator` as VARCHAR(100)) as `Same Assignee & Assignor Indicator`, CAST(`Signed/Order Date` as VARCHAR(100)) as `Signed/Order Date`, CAST(`Special Transaction` as VARCHAR(100)) as `Special Transaction`, CAST(`Street` as VARCHAR(1000)) as `Street`, CAST(`Street Name` as VARCHAR(1000)) as `Street Name`, CAST(`Street in Chinese` as VARCHAR(1000)) as `Street in Chinese`, CAST(`Street in English` as VARCHAR(1000)) as `Street in English`, CAST(`Transaction Remarks` as VARCHAR(1000)) as `Transaction Remarks`, CAST(`Unit` as VARCHAR(100)) as `Unit`, CAST(`Usage` as VARCHAR(100)) as `Usage`, CAST(`View` as VARCHAR(100)) as `View`, CAST(`System Estate ID` as VARCHAR(100)) as `System Estate ID`, CAST(`Age` as VARCHAR(500)) as `Age`, CAST(`Bay Window (square feet)` as VARCHAR(100)) as `Bay Window (square feet)`, CAST(`Efficiency Ratio` as VARCHAR(500)) as `Efficiency Ratio`, CAST(`Gross Area (square meters)` as VARCHAR(500)) as `Gross Area (square meters)`, CAST(`Residual Term` as VARCHAR(500)) as `Residual Term`, CAST(`Saleable Area (square meters)` as VARCHAR(500)) as `Saleable Area (square meters)`, CAST(`Year of Land Premium` as VARCHAR(500)) as `Year of Land Premium` from dfs.`prd_src`.`AAAA_Transaction`
 
2021-09-09 13:16:34,269 [1ec6674f-4541-f8a7-65ee-4a3d5cf63210:foreman] INFO  o.a.d.e.p.s.h.CreateTableHandler - Creating persistent table [AAAA_STG_Transaction].
2021-09-09 13:16:34,444 [1ec6674f-4541-f8a7-65ee-4a3d5cf63210:frag:1:1] INFO  o.a.d.e.w.fragment.FragmentExecutor - 1ec6674f-4541-f8a7-65ee-4a3d5cf63210:1:1: State change requested AWAITING_ALLOCATION --> RUNNING
2021-09-09 13:16:34,444 [1ec6674f-4541-f8a7-65ee-4a3d5cf63210:frag:1:1] INFO  o.a.d.e.w.f.FragmentStatusReporter - 1ec6674f-4541-f8a7-65ee-4a3d5cf63210:1:1: State to report: RUNNING
2021-09-09 13:16:34,460 [1ec6674f-4541-f8a7-65ee-4a3d5cf63210:frag:1:1] WARN  o.a.d.e.e.ExpressionTreeMaterializer - Unable to find value vector of path `Age`, returning null instance.
2021-09-09 13:16:34,460 [1ec6674f-4541-f8a7-65ee-4a3d5cf63210:frag:1:1] WARN  o.a.d.e.e.ExpressionTreeMaterializer - Unable to find value vector of path `Bay Window (square feet)`, returning null instance.
2021-09-09 13:16:34,460 [1ec6674f-4541-f8a7-65ee-4a3d5cf63210:frag:1:1] WARN  o.a.d.e.e.ExpressionTreeMaterializer - Unable to find value vector of path `Efficiency Ratio`, returning null instance.
2021-09-09 13:16:34,461 [1ec6674f-4541-f8a7-65ee-4a3d5cf63210:frag:1:1] WARN  o.a.d.e.e.ExpressionTreeMaterializer - Unable to find value vector of path `Gross Area (square meters)`, returning null instance.
2021-09-09 13:16:34,461 [1ec6674f-4541-f8a7-65ee-4a3d5cf63210:frag:1:1] WARN  o.a.d.e.e.ExpressionTreeMaterializer - Unable to find value vector of path `Residual Term`, returning null instance.
2021-09-09 13:16:34,461 [1ec6674f-4541-f8a7-65ee-4a3d5cf63210:frag:1:1] WARN  o.a.d.e.e.ExpressionTreeMaterializer - Unable to find value vector of path `Saleable Area (square meters)`, returning null instance.
2021-09-09 13:16:34,461 [1ec6674f-4541-f8a7-65ee-4a3d5cf63210:frag:1:1] WARN  o.a.d.e.e.ExpressionTreeMaterializer - Unable to find value vector of path `Year of Land Premium`, returning null instance.
2021-09-09 13:16:34,499 [1ec6674f-4541-f8a7-65ee-4a3d5cf63210:frag:1:1] WARN  o.a.d.e.e.ExpressionTreeMaterializer - Unable to find value vector of path `Year of Land Premium`, returning null instance.
2021-09-09 13:16:34,504 [1ec6674f-4541-f8a7-65ee-4a3d5cf63210:frag:1:1] INFO  o.a.d.e.c.ClassCompilerSelector - Java compiler policy: DEFAULT, Debug option: true
2021-09-09 13:16:35,670 [1ec6674f-4541-f8a7-65ee-4a3d5cf63210:frag:0:0] INFO  o.a.d.e.w.fragment.FragmentExecutor - 1ec6674f-4541-f8a7-65ee-4a3d5cf63210:0:0: State change requested AWAITING_ALLOCATION --> RUNNING
2021-09-09 13:16:35,670 [1ec6674f-4541-f8a7-65ee-4a3d5cf63210:frag:0:0] INFO  o.a.d.e.w.f.FragmentStatusReporter - 1ec6674f-4541-f8a7-65ee-4a3d5cf63210:0:0: State to report: RUNNING
2021-09-09 13:16:37,952 [1ec66757-57c0-8dbb-fc0d-f4bec5e8cef9:frag:1:2] INFO  o.a.d.e.w.fragment.FragmentExecutor - 1ec66757-57c0-8dbb-fc0d-f4bec5e8cef9:1:2: State change requested CANCELLATION_REQUESTED --> FAILED
2021-09-09 13:16:37,956 [1ec66757-57c0-8dbb-fc0d-f4bec5e8cef9:frag:1:2] INFO  o.a.d.e.s.p.ParquetRecordWriter - Aborting writer. Location [/BBBB/PBBBB/tmp/AAAA_STG_Transaction/1_2_0.parquet] on file system [maprfs:///apps/drill] is deleted.
2021-09-09 13:16:37,956 [1ec66757-57c0-8dbb-fc0d-f4bec5e8cef9:frag:1:2] INFO  o.a.d.e.w.fragment.FragmentExecutor - 1ec66757-57c0-8dbb-fc0d-f4bec5e8cef9:1:2: State change requested FAILED --> FINISHED
2021-09-09 13:16:37,956 [1ec66757-57c0-8dbb-fc0d-f4bec5e8cef9:frag:1:2] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: StaleFileException: close() on '/BBBB/PBBBB/tmp/AAAA_STG_Transaction/1_2_0.parquet' failed with error: Stale file handle (116).: retry? true
 
Fragment 1:2
 
[Error Id: e14659a2-8de9-4200-88c8-b79d84567753 on p03:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: StaleFileException: close() on '/BBBB/PBBBB/tmp/AAAA_STG_Transaction/1_2_0.parquet' failed with error: Stale file handle (116).: retry? true
 
Fragment 1:2
 
[Error Id: e14659a2-8de9-4200-88c8-b79d84567753 on p03:31010]
at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633) ~[drill-common-1.14.0-mapr.jar:1.14.0-mapr]
at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:361) [drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:216) [drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:327) [drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.14.0-mapr.jar:1.14.0-mapr]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]
Caused by: com.mapr.fs.StaleFileException: close() on '/BBBB/PBBBB/tmp/AAAA_STG_Transaction/1_2_0.parquet' failed with error: Stale file handle (116).: retry? true
at com.mapr.fs.Inode.checkError(Inode.java:1820) ~[maprfs-6.1.0-mapr.jar:na]
at com.mapr.fs.Inode.checkError(Inode.java:1789) ~[maprfs-6.1.0-mapr.jar:na]
at com.mapr.fs.Inode.close(Inode.java:1082) ~[maprfs-6.1.0-mapr.jar:na]
at com.mapr.fs.MapRFsOutStream.close(MapRFsOutStream.java:225) ~[maprfs-6.1.0-mapr.jar:na]
at com.mapr.fs.MapRFsDataOutputStream.close(MapRFsDataOutputStream.java:50) ~[maprfs-6.1.0-mapr.jar:na]
at org.apache.parquet.hadoop.util.HadoopPositionOutputStream.close(HadoopPositionOutputStream.java:64) ~[parquet-hadoop-1.10.0.jar:1.10.0]
at org.apache.parquet.hadoop.ParquetFileWriter.end(ParquetFileWriter.java:685) ~[parquet-hadoop-1.10.0.jar:1.10.0]
at org.apache.drill.exec.store.parquet.ParquetRecordWriter.flush(ParquetRecordWriter.java:328) ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at org.apache.drill.exec.store.parquet.ParquetRecordWriter.cleanup(ParquetRecordWriter.java:485) ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at org.apache.drill.exec.physical.impl.WriterRecordBatch.closeWriter(WriterRecordBatch.java:191) ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at org.apache.drill.exec.physical.impl.WriterRecordBatch.close(WriterRecordBatch.java:209) ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at org.apache.drill.common.DeferredException.suppressingClose(DeferredException.java:159) ~[drill-common-1.14.0-mapr.jar:1.14.0-mapr]
at org.apache.drill.exec.physical.impl.BaseRootExec.close(BaseRootExec.java:143) ~[drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:377) [drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:211) [drill-java-exec-1.14.0-mapr.jar:1.14.0-mapr]
... 5 common frames omitted
0 Kudos
Reply
Yong-Liu
HPE Pro

тАО09-10-2021 12:33 AM

тАО09-10-2021 12:33 AM

Re: Need help with abnormal Apache Drill Behavior

This error happens  after 3 minutes when the query submitted , right ?

I'm an HPE employee.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
Accept or Kudo
0 Kudos
Reply
CKSunny
Occasional Visitor

тАО09-10-2021 01:52 AM

тАО09-10-2021 01:52 AM

Re: Need help with abnormal Apache Drill Behavior

According to the query profile, I have below informations:

1. "Time" marked in Drill GUI 8047 Profile Page

09/09/2021 13:16:32

2. "Duration" marked in Drill GUI 8047 Profile Page

01 min 6.434 sec

3. "start" marked in Full JSON Profile

09/09/2021 13:16:32:108

4. "end" marked in Full JSON Profile

09/09/2021 13:17:38:542

5. "planEnd" marked in Full JSON Profile

09/09/2021 13:16:34:317

6. "queueWaitEnd" marked in Full JSON Profile

09/09/2021 13:16:34:320

I can include the Full JSON Profile if needed.

0 Kudos
Reply
Yong-Liu
HPE Pro

тАО09-10-2021 02:25 AM

тАО09-10-2021 02:25 AM

Re: Need help with abnormal Apache Drill Behavior

Yes, please upload the full query profile. and another question ,why you are using old drill 1.14 instead of 1.16?

I'm an HPE employee.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
Accept or Kudo
0 Kudos
Reply
CKSunny
Occasional Visitor

тАО09-10-2021 02:40 AM - edited тАО09-10-2021 02:43 AM

тАО09-10-2021 02:40 AM - edited тАО09-10-2021 02:43 AM

Re: Need help with abnormal Apache Drill Behavior

This is still a POC environment setup two years ago and there will not be plans for upgrade if this issue is proved cannot be fixed.

Below is the Full Detail, I have removed a large set of file name and masked some hostname and filename:

 

{
    "id": {
        "part1": 2217573456688576800,
        "part2": 7344889669700957000
    },
    "type": 1,
    "start": 1631164592108,
    "end": 1631164658542,
    "query": "create table dfs.`prd_tmp`.`AAAA_STG_Transaction` as select CAST(`Block` as VARCHAR(100)) as `Block`, CAST(`Building Name` as VARCHAR(1000)) as `Building Name`, CAST(`Building Name in Chinese` as VARCHAR(500)) as `Building Name in Chinese`, CAST(`Building Name in English` as VARCHAR(500)) as `Building Name in English`, CAST(`Building Remarks` as VARCHAR(3000)) as `Building Remarks`, CAST(`Consideration Remarks` as VARCHAR(3000)) as `Consideration Remarks`, CAST(`DD/Lot No.` as VARCHAR(500)) as `DD/Lot No.`, CAST(`Data Source` as VARCHAR(100)) as `Data Source`, CAST(`Delivery Date` as VARCHAR(100)) as `Delivery Date`, CAST(`Direction` as VARCHAR(100)) as `Direction`, CAST(`District` as VARCHAR(100)) as `District`, CAST(`District 1` as VARCHAR(100)) as `District 1`, CAST(`District 1 in Chinese` as VARCHAR(100)) as `District 1 in Chinese`, CAST(`District 1 in English` as VARCHAR(100)) as `District 1 in English`, CAST(`District 2` as VARCHAR(100)) as `District 2`, CAST(`District 2 in Chinese` as VARCHAR(100)) as `District 2 in Chinese`, CAST(`District 2 in English` as VARCHAR(100)) as `District 2 in English`, CAST(`Estate Name` as VARCHAR(1000)) as `Estate Name`, CAST(`Estate Name in Chinese` as VARCHAR(500)) as `Estate Name in Chinese`, CAST(`Estate Name in English` as VARCHAR(500)) as `Estate Name in English`, CAST(`Facilities` as VARCHAR(1000)) as `Facilities`, CAST(`Floor` as VARCHAR(100)) as `Floor`, CAST(`Floor Plan` as VARCHAR(500)) as `Floor Plan`, CAST(`Gross Area (square feet)` as VARCHAR(500)) as `Gross Area (square feet)`, CAST(`Gross Unit Price ($/square foot)` as VARCHAR(500)) as `Gross Unit Price ($/square foot)`, CAST(`Land Premium` as VARCHAR(100)) as `Land Premium`, CAST(`Lease Term` as VARCHAR(100)) as `Lease Term`, CAST(`Memo No.` as VARCHAR(100)) as `Memo No.`, CAST(`Nature` as VARCHAR(100)) as `Nature`, CAST(`Occupation Date` as VARCHAR(100)) as `Occupation Date`, CAST(`Phase` as VARCHAR(100)) as `Phase`, CAST(`Price ($)` as VARCHAR(500)) as `Price ($)`, CAST(`Property No (PRN)` as VARCHAR(1000)) as `Property No (PRN)`, CAST(`Property Type` as VARCHAR(100)) as `Property Type`, CAST(`Property Type Detail` as VARCHAR(100)) as `Property Type Detail`, CAST(`Region` as VARCHAR(100)) as `Region`, CAST(`Saleable Area (square feet)` as VARCHAR(500)) as `Saleable Area (square feet)`, CAST(`Saleable Unit Price ($/square foot)` as VARCHAR(500)) as `Saleable Unit Price ($/square foot)`, CAST(`Same Assignee & Assignor Indicator` as VARCHAR(100)) as `Same Assignee & Assignor Indicator`, CAST(`Signed/Order Date` as VARCHAR(100)) as `Signed/Order Date`, CAST(`Special Transaction` as VARCHAR(100)) as `Special Transaction`, CAST(`Street` as VARCHAR(1000)) as `Street`, CAST(`Street Name` as VARCHAR(1000)) as `Street Name`, CAST(`Street in Chinese` as VARCHAR(1000)) as `Street in Chinese`, CAST(`Street in English` as VARCHAR(1000)) as `Street in English`, CAST(`Transaction Remarks` as VARCHAR(1000)) as `Transaction Remarks`, CAST(`Unit` as VARCHAR(100)) as `Unit`, CAST(`Usage` as VARCHAR(100)) as `Usage`, CAST(`View` as VARCHAR(100)) as `View`, CAST(`System Estate ID` as VARCHAR(100)) as `System Estate ID`, CAST(`Age` as VARCHAR(500)) as `Age`, CAST(`Bay Window (square feet)` as VARCHAR(100)) as `Bay Window (square feet)`, CAST(`Efficiency Ratio` as VARCHAR(500)) as `Efficiency Ratio`, CAST(`Gross Area (square meters)` as VARCHAR(500)) as `Gross Area (square meters)`, CAST(`Residual Term` as VARCHAR(500)) as `Residual Term`, CAST(`Saleable Area (square meters)` as VARCHAR(500)) as `Saleable Area (square meters)`, CAST(`Year of Land Premium` as VARCHAR(500)) as `Year of Land Premium` from dfs.`prd_src`.`AAAA_Transaction`\r\n",
    "plan": "00-00    Screen : rowType = RecordType(VARCHAR(255) Fragment, BIGINT Number of records written): rowcount = 353289.0, cumulative cost = {2155062.9 rows, 1.247463459E8 cpu, 3.61768124E8 io, 2.894143488E9 network, 0.0 memory}, id = 82615\n00-01      Project(Fragment=[$0], Number of records written=[$1]) : rowType = RecordType(VARCHAR(255) Fragment, BIGINT Number of records written): rowcount = 353289.0, cumulative cost = {2119734.0 rows, 1.24711017E8 cpu, 3.61768124E8 io, 2.894143488E9 network, 0.0 memory}, id = 82614\n00-02        UnionExchange : rowType = RecordType(VARCHAR(255) Fragment, BIGINT Number of records written): rowcount = 353289.0, cumulative cost = {1766445.0 rows, 1.24004439E8 cpu, 3.61768124E8 io, 2.894143488E9 network, 0.0 memory}, id = 82613\n01-01          Writer : rowType = RecordType(VARCHAR(255) Fragment, BIGINT Number of records written): rowcount = 353289.0, cumulative cost = {1413156.0 rows, 1.21178127E8 cpu, 3.61768124E8 io, 0.0 network, 0.0 memory}, id = 82612\n01-02            ProjectAllowDup(Block=[$0], Building Name=[$1], Building Name in Chinese=[$2], Building Name in English=[$3], Building Remarks=[$4], Consideration Remarks=[$5], DD/Lot No.=[$6], Data Source=[$7], Delivery Date=[$8], Direction=[$9], District=[$10], District 1=[$11], District 1 in Chinese=[$12], District 1 in English=[$13], District 2=[$14], District 2 in Chinese=[$15], District 2 in English=[$16], Estate Name=[$17], Estate Name in Chinese=[$18], Estate Name in English=[$19], Facilities=[$20], Floor=[$21], Floor Plan=[$22], Gross Area (square feet)=[$23], Gross Unit Price ($/square foot)=[$24], Land Premium=[$25], Lease Term=[$26], Memo No.=[$27], Nature=[$28], Occupation Date=[$29], Phase=[$30], Price ($)=[$31], Property No (PRN)=[$32], Property Type=[$33], Property Type Detail=[$34], Region=[$35], Saleable Area (square feet)=[$36], Saleable Unit Price ($/square foot)=[$37], Same Assignee & Assignor Indicator=[$38], Signed/Order Date=[$39], Special Transaction=[$40], Street=[$41], Street Name=[$42], Street in Chinese=[$43], Street in English=[$44], Transaction Remarks=[$45], Unit=[$46], Usage=[$47], View=[$48], System Estate ID=[$49], Age=[$50], Bay Window (square feet)=[$51], Efficiency Ratio=[$52], Gross Area (square meters)=[$53], Residual Term=[$54], Saleable Area (square meters)=[$55], Year of Land Premium=[$56]) : rowType = RecordType(VARCHAR(100) Block, VARCHAR(1000) Building Name, VARCHAR(500) Building Name in Chinese, VARCHAR(500) Building Name in English, VARCHAR(3000) Building Remarks, VARCHAR(3000) Consideration Remarks, VARCHAR(500) DD/Lot No., VARCHAR(100) Data Source, VARCHAR(100) Delivery Date, VARCHAR(100) Direction, VARCHAR(100) District, VARCHAR(100) District 1, VARCHAR(100) District 1 in Chinese, VARCHAR(100) District 1 in English, VARCHAR(100) District 2, VARCHAR(100) District 2 in Chinese, VARCHAR(100) District 2 in English, VARCHAR(1000) Estate Name, VARCHAR(500) Estate Name in Chinese, VARCHAR(500) Estate Name in English, VARCHAR(1000) Facilities, VARCHAR(100) Floor, VARCHAR(500) Floor Plan, VARCHAR(500) Gross Area (square feet), VARCHAR(500) Gross Unit Price ($/square foot), VARCHAR(100) Land Premium, VARCHAR(100) Lease Term, VARCHAR(100) Memo No., VARCHAR(100) Nature, VARCHAR(100) Occupation Date, VARCHAR(100) Phase, VARCHAR(500) Price ($), VARCHAR(1000) Property No (PRN), VARCHAR(100) Property Type, VARCHAR(100) Property Type Detail, VARCHAR(100) Region, VARCHAR(500) Saleable Area (square feet), VARCHAR(500) Saleable Unit Price ($/square foot), VARCHAR(100) Same Assignee & Assignor Indicator, VARCHAR(100) Signed/Order Date, VARCHAR(100) Special Transaction, VARCHAR(1000) Street, VARCHAR(1000) Street Name, VARCHAR(1000) Street in Chinese, VARCHAR(1000) Street in English, VARCHAR(1000) Transaction Remarks, VARCHAR(100) Unit, VARCHAR(100) Usage, VARCHAR(100) View, VARCHAR(100) System Estate ID, VARCHAR(500) Age, VARCHAR(100) Bay Window (square feet), VARCHAR(500) Efficiency Ratio, VARCHAR(500) Gross Area (square meters), VARCHAR(500) Residual Term, VARCHAR(500) Saleable Area (square meters), VARCHAR(500) Year of Land Premium): rowcount = 353289.0, cumulative cost = {1059867.0 rows, 1.20824838E8 cpu, 3.61768124E8 io, 0.0 network, 0.0 memory}, id = 82611\n01-03              Project(Block=[CAST($0):VARCHAR(100) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Building Name=[CAST($1):VARCHAR(1000) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Building Name in Chinese=[CAST($2):VARCHAR(500) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Building Name in English=[CAST($3):VARCHAR(500) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Building Remarks=[CAST($4):VARCHAR(3000) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Consideration Remarks=[CAST($5):VARCHAR(3000) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], DD/Lot No.=[CAST($6):VARCHAR(500) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Data Source=[CAST($7):VARCHAR(100) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Delivery Date=[CAST($8):VARCHAR(100) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Direction=[CAST($9):VARCHAR(100) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], District=[CAST($10):VARCHAR(100) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], District 1=[CAST($11):VARCHAR(100) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], District 1 in Chinese=[CAST($12):VARCHAR(100) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], District 1 in English=[CAST($13):VARCHAR(100) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], District 2=[CAST($14):VARCHAR(100) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], District 2 in Chinese=[CAST($15):VARCHAR(100) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], District 2 in English=[CAST($16):VARCHAR(100) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Estate Name=[CAST($17):VARCHAR(1000) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Estate Name in Chinese=[CAST($18):VARCHAR(500) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Estate Name in English=[CAST($19):VARCHAR(500) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Facilities=[CAST($20):VARCHAR(1000) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Floor=[CAST($21):VARCHAR(100) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Floor Plan=[CAST($22):VARCHAR(500) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Gross Area (square feet)=[CAST($23):VARCHAR(500) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Gross Unit Price ($/square foot)=[CAST($24):VARCHAR(500) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Land Premium=[CAST($25):VARCHAR(100) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Lease Term=[CAST($26):VARCHAR(100) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Memo No.=[CAST($27):VARCHAR(100) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Nature=[CAST($28):VARCHAR(100) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Occupation Date=[CAST($29):VARCHAR(100) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Phase=[CAST($30):VARCHAR(100) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Price ($)=[CAST($31):VARCHAR(500) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Property No (PRN)=[CAST($32):VARCHAR(1000) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Property Type=[CAST($33):VARCHAR(100) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Property Type Detail=[CAST($34):VARCHAR(100) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Region=[CAST($35):VARCHAR(100) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Saleable Area (square feet)=[CAST($36):VARCHAR(500) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Saleable Unit Price ($/square foot)=[CAST($37):VARCHAR(500) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Same Assignee & Assignor Indicator=[CAST($38):VARCHAR(100) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Signed/Order Date=[CAST($39):VARCHAR(100) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Special Transaction=[CAST($40):VARCHAR(100) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Street=[CAST($41):VARCHAR(1000) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Street Name=[CAST($42):VARCHAR(1000) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Street in Chinese=[CAST($43):VARCHAR(1000) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Street in English=[CAST($44):VARCHAR(1000) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Transaction Remarks=[CAST($45):VARCHAR(1000) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Unit=[CAST($46):VARCHAR(100) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Usage=[CAST($47):VARCHAR(100) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], View=[CAST($48):VARCHAR(100) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], System Estate ID=[CAST($49):VARCHAR(100) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Age=[CAST($50):VARCHAR(500) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Bay Window (square feet)=[CAST($51):VARCHAR(100) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Efficiency Ratio=[CAST($52):VARCHAR(500) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Gross Area (square meters)=[CAST($53):VARCHAR(500) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Residual Term=[CAST($54):VARCHAR(500) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Saleable Area (square meters)=[CAST($55):VARCHAR(500) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"], Year of Land Premium=[CAST($56):VARCHAR(500) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\"]) : rowType = RecordType(VARCHAR(100) Block, VARCHAR(1000) Building Name, VARCHAR(500) Building Name in Chinese, VARCHAR(500) Building Name in English, VARCHAR(3000) Building Remarks, VARCHAR(3000) Consideration Remarks, VARCHAR(500) DD/Lot No., VARCHAR(100) Data Source, VARCHAR(100) Delivery Date, VARCHAR(100) Direction, VARCHAR(100) District, VARCHAR(100) District 1, VARCHAR(100) District 1 in Chinese, VARCHAR(100) District 1 in English, VARCHAR(100) District 2, VARCHAR(100) District 2 in Chinese, VARCHAR(100) District 2 in English, VARCHAR(1000) Estate Name, VARCHAR(500) Estate Name in Chinese, VARCHAR(500) Estate Name in English, VARCHAR(1000) Facilities, VARCHAR(100) Floor, VARCHAR(500) Floor Plan, VARCHAR(500) Gross Area (square feet), VARCHAR(500) Gross Unit Price ($/square foot), VARCHAR(100) Land Premium, VARCHAR(100) Lease Term, VARCHAR(100) Memo No., VARCHAR(100) Nature, VARCHAR(100) Occupation Date, VARCHAR(100) Phase, VARCHAR(500) Price ($), VARCHAR(1000) Property No (PRN), VARCHAR(100) Property Type, VARCHAR(100) Property Type Detail, VARCHAR(100) Region, VARCHAR(500) Saleable Area (square feet), VARCHAR(500) Saleable Unit Price ($/square foot), VARCHAR(100) Same Assignee & Assignor Indicator, VARCHAR(100) Signed/Order Date, VARCHAR(100) Special Transaction, VARCHAR(1000) Street, VARCHAR(1000) Street Name, VARCHAR(1000) Street in Chinese, VARCHAR(1000) Street in English, VARCHAR(1000) Transaction Remarks, VARCHAR(100) Unit, VARCHAR(100) Usage, VARCHAR(100) View, VARCHAR(100) System Estate ID, VARCHAR(500) Age, VARCHAR(100) Bay Window (square feet), VARCHAR(500) Efficiency Ratio, VARCHAR(500) Gross Area (square meters), VARCHAR(500) Residual Term, VARCHAR(500) Saleable Area (square meters), VARCHAR(500) Year of Land Premium): rowcount = 353289.0, cumulative cost = {706578.0 rows, 1.00687365E8 cpu, 3.61768124E8 io, 0.0 network, 0.0 memory}, id = 82610\n01-04                Scan(table=[[dfs, prd_src, AAAA_Transaction]], groupscan=[EasyGroupScan [selectionRoot=maprfs:/BBBB/PBBBB/cleansed/AAAA_Transaction, numFiles=16377, columns=[`Block`, `Building Name`, `Building Name in Chinese`, `Building Name in English`, `Building Remarks`, `Consideration Remarks`, `DD/Lot No.`, `Data Source`, `Delivery Date`, `Direction`, `District`, `District 1`, `District 1 in Chinese`, `District 1 in English`, `District 2`, `District 2 in Chinese`, `District 2 in English`, `Estate Name`, `Estate Name in Chinese`, `Estate Name in English`, `Facilities`, `Floor`, `Floor Plan`, `Gross Area (square feet)`, `Gross Unit Price ($/square foot)`, `Land Premium`, `Lease Term`, `Memo No.`, `Nature`, `Occupation Date`, `Phase`, `Price ($)`, `Property No (PRN)`, `Property Type`, `Property Type Detail`, `Region`, `Saleable Area (square feet)`, `Saleable Unit Price ($/square foot)`, `Same Assignee & Assignor Indicator`, `Signed/Order Date`, `Special Transaction`, `Street`, `Street Name`, `Street in Chinese`, `Street in English`, `Transaction Remarks`, `Unit`, `Usage`, `View`, `System Estate ID`, `Age`, `Bay Window (square feet)`, `Efficiency Ratio`, `Gross Area (square meters)`, `Residual Term`, `Saleable Area (square meters)`, `Year of Land Premium`], files=[many_other_files, maprfs:///BBBB/PBBBB/cleansed/AAAA_Transaction/Patched_Sys_Id_T-30_Cleansed_txn_NT_08102019_0711201918351573122946.json, maprfs:///BBBB/PBBBB/cleansed/AAAA_Transaction/Patched_Sys_Id_Cleansed_txn_KL_09032021_1203202108231615508582.json]]]) : rowType = RecordType(ANY Block, ANY Building Name, ANY Building Name in Chinese, ANY Building Name in English, ANY Building Remarks, ANY Consideration Remarks, ANY DD/Lot No., ANY Data Source, ANY Delivery Date, ANY Direction, ANY District, ANY District 1, ANY District 1 in Chinese, ANY District 1 in English, ANY District 2, ANY District 2 in Chinese, ANY District 2 in English, ANY Estate Name, ANY Estate Name in Chinese, ANY Estate Name in English, ANY Facilities, ANY Floor, ANY Floor Plan, ANY Gross Area (square feet), ANY Gross Unit Price ($/square foot), ANY Land Premium, ANY Lease Term, ANY Memo No., ANY Nature, ANY Occupation Date, ANY Phase, ANY Price ($), ANY Property No (PRN), ANY Property Type, ANY Property Type Detail, ANY Region, ANY Saleable Area (square feet), ANY Saleable Unit Price ($/square foot), ANY Same Assignee & Assignor Indicator, ANY Signed/Order Date, ANY Special Transaction, ANY Street, ANY Street Name, ANY Street in Chinese, ANY Street in English, ANY Transaction Remarks, ANY Unit, ANY Usage, ANY View, ANY System Estate ID, ANY Age, ANY Bay Window (square feet), ANY Efficiency Ratio, ANY Gross Area (square meters), ANY Residual Term, ANY Saleable Area (square meters), ANY Year of Land Premium): rowcount = 353289.0, cumulative cost = {353289.0 rows, 2.0137473E7 cpu, 3.61768124E8 io, 0.0 network, 0.0 memory}, id = 82609\n",
    "foreman": {
        "address": "p03",
        "userPort": 31010,
        "controlPort": 31011,
        "dataPort": 31012,
        "version": "1.14.0-mapr",
        "state": 0
    },
    "state": 4,
    "totalFragments": 5,
    "finishedFragments": 3,
    "fragmentProfile": [
        {
            "majorFragmentId": 0,
            "minorFragmentProfile": [
                {
                    "state": 6,
                    "minorFragmentId": 0,
                    "operatorProfile": [
                        {
                            "inputProfile": [
                                {
                                    "records": 4,
                                    "batches": 4,
                                    "schemas": 1
                                }
                            ],
                            "operatorId": 2,
                            "operatorType": 11,
                            "setupNanos": 0,
                            "processNanos": 233054,
                            "peakLocalMemoryAllocated": 256,
                            "metric": [
                                {
                                    "metricId": 0,
                                    "longValue": 0
                                },
                                {
                                    "metricId": 1,
                                    "longValue": 4
                                }
                            ],
                            "waitNanos": 62863147126
                        },
                        {
                            "inputProfile": [
                                {
                                    "records": 4,
                                    "batches": 4,
                                    "schemas": 1
                                }
                            ],
                            "operatorId": 1,
                            "operatorType": 10,
                            "setupNanos": 254296,
                            "processNanos": 165089,
                            "peakLocalMemoryAllocated": 256,
                            "waitNanos": 0
                        },
                        {
                            "inputProfile": [
                                {
                                    "records": 4,
                                    "batches": 4,
                                    "schemas": 1
                                }
                            ],
                            "operatorId": 0,
                            "operatorType": 13,
                            "setupNanos": 0,
                            "processNanos": 83463,
                            "peakLocalMemoryAllocated": 256,
                            "metric": [
                                {
                                    "metricId": 0,
                                    "longValue": 76
                                }
                            ],
                            "waitNanos": 506997
                        }
                    ],
                    "startTime": 1631164594323,
                    "endTime": 1631164658542,
                    "memoryUsed": 3000000,
                    "maxMemoryUsed": 3000256,
                    "endpoint": {
                        "address": "p03",
                        "userPort": 31010,
                        "controlPort": 31011,
                        "dataPort": 31012,
                        "version": "1.14.0-mapr",
                        "state": 0
                    },
                    "lastUpdate": 1631164658542,
                    "lastProgress": 1631164658542
                }
            ]
        },
        {
            "majorFragmentId": 1,
            "minorFragmentProfile": [
                {
                    "state": 3,
                    "minorFragmentId": 0,
                    "operatorProfile": [
                        {
                            "inputProfile": [
                                {
                                    "records": 60211,
                                    "batches": 6832,
                                    "schemas": 5
                                }
                            ],
                            "operatorId": 0,
                            "operatorType": 29,
                            "setupNanos": 0,
                            "processNanos": 7607111448,
                            "peakLocalMemoryAllocated": 4706304,
                            "waitNanos": 718499120
                        },
                        {
                            "inputProfile": [
                                {
                                    "records": 60211,
                                    "batches": 2737,
                                    "schemas": 5
                                }
                            ],
                            "operatorId": 3,
                            "operatorType": 10,
                            "setupNanos": 76365986,
                            "processNanos": 2604966353,
                            "peakLocalMemoryAllocated": 4575232,
                            "waitNanos": 0
                        },
                        {
                            "inputProfile": [
                                {
                                    "records": 60211,
                                    "batches": 2737,
                                    "schemas": 1
                                }
                            ],
                            "operatorId": 2,
                            "operatorType": 10,
                            "setupNanos": 478467,
                            "processNanos": 527016559,
                            "peakLocalMemoryAllocated": 4509696,
                            "waitNanos": 0
                        },
                        {
                            "inputProfile": [
                                {
                                    "records": 60211,
                                    "batches": 2737,
                                    "schemas": 1
                                }
                            ],
                            "operatorId": 1,
                            "operatorType": 25,
                            "setupNanos": 541813,
                            "processNanos": 1515029202,
                            "peakLocalMemoryAllocated": 80,
                            "waitNanos": 0
                        },
                        {
                            "inputProfile": [
                                {
                                    "records": 1,
                                    "batches": 1,
                                    "schemas": 1
                                }
                            ],
                            "operatorId": 0,
                            "operatorType": 0,
                            "setupNanos": 0,
                            "processNanos": 35148,
                            "peakLocalMemoryAllocated": 80,
                            "metric": [
                                {
                                    "metricId": 0,
                                    "longValue": 19
                                }
                            ],
                            "waitNanos": 1460465
                        }
                    ],
                    "startTime": 1631164596527,
                    "endTime": 1631164609719,
                    "memoryUsed": 0,
                    "maxMemoryUsed": 63722752,
                    "endpoint": {
                        "address": "p02",
                        "userPort": 31010,
                        "controlPort": 31011,
                        "dataPort": 31012,
                        "version": "1.14.0-mapr",
                        "state": 1,
                        "httpPort": 8047
                    },
                    "lastUpdate": 1631164607521,
                    "lastProgress": 1631164607521
                },
                {
                    "state": 3,
                    "minorFragmentId": 3,
                    "operatorProfile": [
                        {
                            "inputProfile": [
                                {
                                    "records": 0,
                                    "batches": 4094,
                                    "schemas": 0
                                }
                            ],
                            "operatorId": 0,
                            "operatorType": 29,
                            "setupNanos": 0,
                            "processNanos": 1208788221,
                            "peakLocalMemoryAllocated": 0,
                            "waitNanos": 1353097
                        },
                        {
                            "inputProfile": [
                                {
                                    "records": 0,
                                    "batches": 0,
                                    "schemas": 0
                                }
                            ],
                            "operatorId": 3,
                            "operatorType": 10,
                            "setupNanos": 0,
                            "processNanos": 10346,
                            "peakLocalMemoryAllocated": 0,
                            "waitNanos": 0
                        },
                        {
                            "inputProfile": [
                                {
                                    "records": 0,
                                    "batches": 0,
                                    "schemas": 0
                                }
                            ],
                            "operatorId": 2,
                            "operatorType": 10,
                            "setupNanos": 0,
                            "processNanos": 2478,
                            "peakLocalMemoryAllocated": 0,
                            "waitNanos": 0
                        },
                        {
                            "inputProfile": [
                                {
                                    "records": 0,
                                    "batches": 0,
                                    "schemas": 0
                                }
                            ],
                            "operatorId": 1,
                            "operatorType": 25,
                            "setupNanos": 66686,
                            "processNanos": 68934,
                            "peakLocalMemoryAllocated": 80,
                            "waitNanos": 0
                        },
                        {
                            "inputProfile": [
                                {
                                    "records": 1,
                                    "batches": 1,
                                    "schemas": 1
                                }
                            ],
                            "operatorId": 0,
                            "operatorType": 0,
                            "setupNanos": 0,
                            "processNanos": 37853,
                            "peakLocalMemoryAllocated": 80,
                            "metric": [
                                {
                                    "metricId": 0,
                                    "longValue": 19
                                }
                            ],
                            "waitNanos": 717109
                        }
                    ],
                    "startTime": 1631164596583,
                    "endTime": 1631164597927,
                    "memoryUsed": 0,
                    "maxMemoryUsed": 7048064,
                    "endpoint": {
                        "address": "p04",
                        "userPort": 31010,
                        "controlPort": 31011,
                        "dataPort": 31012,
                        "version": "1.14.0-mapr",
                        "state": 1,
                        "httpPort": 8047
                    },
                    "lastUpdate": 1631164595674,
                    "lastProgress": 1631164595674
                },
                {
                    "state": 3,
                    "minorFragmentId": 1,
                    "operatorProfile": [
                        {
                            "inputProfile": [
                                {
                                    "records": 74761,
                                    "batches": 7213,
                                    "schemas": 2
                                }
                            ],
                            "operatorId": 0,
                            "operatorType": 29,
                            "setupNanos": 0,
                            "processNanos": 8568752452,
                            "peakLocalMemoryAllocated": 2662400,
                            "waitNanos": 798916572
                        },
                        {
                            "inputProfile": [
                                {
                                    "records": 74761,
                                    "batches": 3119,
                                    "schemas": 2
                                }
                            ],
                            "operatorId": 3,
                            "operatorType": 10,
                            "setupNanos": 13501576809,
                            "processNanos": 3166642499,
                            "peakLocalMemoryAllocated": 3035136,
                            "waitNanos": 0
                        },
                        {
                            "inputProfile": [
                                {
                                    "records": 74761,
                                    "batches": 3119,
                                    "schemas": 1
                                }
                            ],
                            "operatorId": 2,
                            "operatorType": 10,
                            "setupNanos": 612990,
                            "processNanos": 691797582,
                            "peakLocalMemoryAllocated": 3035136,
                            "waitNanos": 0
                        },
                        {
                            "inputProfile": [
                                {
                                    "records": 74761,
                                    "batches": 3119,
                                    "schemas": 1
                                }
                            ],
                            "operatorId": 1,
                            "operatorType": 25,
                            "setupNanos": 1234405,
                            "processNanos": 2346444849,
                            "peakLocalMemoryAllocated": 80,
                            "waitNanos": 0
                        },
                        {
                            "inputProfile": [
                                {
                                    "records": 1,
                                    "batches": 1,
                                    "schemas": 1
                                }
                            ],
                            "operatorId": 0,
                            "operatorType": 0,
                            "setupNanos": 0,
                            "processNanos": 51497,
                            "peakLocalMemoryAllocated": 80,
                            "metric": [
                                {
                                    "metricId": 0,
                                    "longValue": 19
                                }
                            ],
                            "waitNanos": 703794
                        }
                    ],
                    "startTime": 1631164594323,
                    "endTime": 1631164623528,
                    "memoryUsed": 0,
                    "maxMemoryUsed": 75080704,
                    "endpoint": {
                        "address": "p03",
                        "userPort": 31010,
                        "controlPort": 31011,
                        "dataPort": 31012,
                        "version": "1.14.0-mapr",
                        "state": 1,
                        "httpPort": 8047
                    },
                    "lastUpdate": 1631164623528,
                    "lastProgress": 1631164623528
                },
                {
                    "state": 2,
                    "minorFragmentId": 2,
                    "operatorProfile": [
                        {
                            "inputProfile": [
                                {
                                    "records": 57292,
                                    "batches": 4667,
                                    "schemas": 6
                                }
                            ],
                            "operatorId": 0,
                            "operatorType": 29,
                            "setupNanos": 0,
                            "processNanos": 5580388734,
                            "peakLocalMemoryAllocated": 2715648,
                            "waitNanos": 625827125
                        },
                        {
                            "inputProfile": [
                                {
                                    "records": 57292,
                                    "batches": 2334,
                                    "schemas": 6
                                }
                            ],
                            "operatorId": 3,
                            "operatorType": 10,
                            "setupNanos": 52510868259,
                            "processNanos": 2281776320,
                            "peakLocalMemoryAllocated": 3035136,
                            "waitNanos": 0
                        },
                        {
                            "inputProfile": [
                                {
                                    "records": 57102,
                                    "batches": 2333,
                                    "schemas": 1
                                }
                            ],
                            "operatorId": 2,
                            "operatorType": 10,
                            "setupNanos": 536785,
                            "processNanos": 476569131,
                            "peakLocalMemoryAllocated": 3035136,
                            "waitNanos": 0
                        },
                        {
                            "inputProfile": [
                                {
                                    "records": 57102,
                                    "batches": 2333,
                                    "schemas": 1
                                }
                            ],
                            "operatorId": 1,
                            "operatorType": 25,
                            "setupNanos": 552893,
                            "processNanos": 1190891018,
                            "peakLocalMemoryAllocated": 0,
                            "waitNanos": 0
                        },
                        {
                            "inputProfile": [
                                {
                                    "records": 0,
                                    "batches": 0,
                                    "schemas": 0
                                }
                            ],
                            "operatorId": 0,
                            "operatorType": 0,
                            "setupNanos": 0,
                            "processNanos": 774,
                            "peakLocalMemoryAllocated": 0,
                            "waitNanos": 0
                        }
                    ],
                    "startTime": 1631164561999,
                    "endTime": 1631164624801,
                    "memoryUsed": 59387328,
                    "maxMemoryUsed": 59387328,
                    "endpoint": {
                        "address": "p01",
                        "userPort": 31010,
                        "controlPort": 31011,
                        "dataPort": 31012,
                        "version": "1.14.0-mapr",
                        "state": 1,
                        "httpPort": 8047
                    },
                    "lastUpdate": 1631164657134,
                    "lastProgress": 1631164657134
                }
            ]
        }
    ],
    "user": "anonymous",
    "error": "SYSTEM ERROR: Drill Remote Exception\n\n",
    "verboseError": "SYSTEM ERROR: Drill Remote Exception\n\n\n\n",
    "errorId": "e6bc1d89-b320-4934-96be-a7e11aad5242",
    "errorNode": "p03:31010",
    "optionsJson": "[ ]",
    "planEnd": 1631164594317,
    "queueWaitEnd": 1631164594320,
    "totalCost": 2473023,
    "queueName": "Unknown",
    "queryId": "1ec6674f-4541-f8a7-65ee-4a3d5cf63210"
}

 

 

0 Kudos
Reply
The opinions expressed above are the personal opinions of the authors, not of Hewlett Packard Enterprise. By using this site, you accept the Terms of Use and Rules of Participation.